sparkforensics-server 0.5.0 → 0.5.2

package/public/docs/assets/user-guide_mcp-tools_evaluate-budgets.md.BB0nDY3Z.js

@@ -0,0 +1,26 @@
+import{_ as s,o as i,c as a,a5 as t}from"./chunks/framework.DSg0KOwT.js";const k=JSON.parse('{"title":"evaluate_budgets","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/evaluate-budgets.md","filePath":"user-guide/mcp-tools/evaluate-budgets.md"}'),n={name:"user-guide/mcp-tools/evaluate-budgets.md"};function o(d,e,c,l,r,h){return i(),a("div",null,[...e[0]||(e[0]=[t(`<h1 id="evaluate-budgets" tabindex="-1"><code>evaluate_budgets</code> <a class="header-anchor" href="#evaluate-budgets" aria-label="Permalink to &quot;\`evaluate_budgets\`&quot;">​</a></h1><p>Reference for the <code>evaluate_budgets</code> tool of the SparkForensics MCP server.</p><p>Evaluate a run against pass/fail thresholds: the MCP equivalent of the CLI&#39;s <code>--max-runtime</code>/<code>--max-spill</code>/etc. budget flags, thin-wrapped over the same <code>evaluateBudgets()</code> the <code>sparkforensics-analyze</code> CLI uses.</p><p>Parameters (all optional):</p><ul><li><code>source</code> / <code>runId</code>: identify the run to evaluate (same shape as <code>diagnose_run</code>). When <code>runIdB</code>/<code>sourceB</code> is also given, this run is the regression baseline instead.</li><li><code>maxRuntimeMs</code>, <code>maxSpillGb</code>, <code>maxSkewRatio</code>, <code>maxFailedTaskRatePct</code>, <code>minEfficiencyPct</code>: absolute budgets, each evaluated only if provided. With two runs they apply to the candidate (<code>runIdB</code>/<code>sourceB</code>), the same as the CLI&#39;s <code>--baseline</code> mode. <code>minEfficiencyPct</code> measures busy core time, the share of executor core time that ran tasks (100 minus the dashboard&#39;s Unused core time), not the dashboard&#39;s Efficiency tile. <code>maxFailedTaskRatePct</code> reads the task failure rate off the <code>jobFailureRate</code> finding, so when that finding didn&#39;t fire (under 10% of jobs failed, by default) it passes whatever the task failure rate.</li><li><code>runIdB</code> / <code>sourceB</code>: a candidate run to compare against the first, so regression budgets can be evaluated. Same <code>runId</code>/<code>source</code> shape, resolved the same way. Omit both to skip regression budgets.</li><li><code>maxRegressionPct</code>, <code>regressionMetric</code>: fail if <code>regressionMetric</code> (default <code>wallClock</code>; see <a href="./../getting-started/regression-metric-keys.html#regression-metric-keys">Regression metric keys</a> for the full list) regressed by more than <code>maxRegressionPct</code>% between the first run (baseline) and the second run (candidate). Requires <code>runIdB</code>/<code>sourceB</code>. <code>regressionMetric</code> without <code>maxRegressionPct</code> fails the call. A volume key (<code>inputBytes</code>, <code>outputBytes</code>, <code>taskCount</code>, <code>executorsAdded</code>), which has no regression direction, or an unknown key reports <code>inconclusive</code>.</li><li><code>regressionBudgets</code>: further regression budgets, an array of <code>{ &quot;metric&quot;: &quot;&lt;key&gt;&quot;, &quot;maxPct&quot;: &lt;number&gt; }</code>, the MCP form of the CLI&#39;s repeated <code>--regression-budget &lt;metric&gt;:&lt;pct&gt;</code>. Each is checked like <code>maxRegressionPct</code> and reports its own <code>max-regression</code> result. A metric can be budgeted once across this list and the <code>maxRegressionPct</code>/<code>regressionMetric</code> pair (which counts as one budget); a repeat fails the call. Requires <code>runIdB</code>/<code>sourceB</code>, as <code>maxRegressionPct</code> does.</li><li><code>stageRegressionBudgets</code>: per paired stage budgets, an array of <code>{ &quot;metric&quot;: &quot;&lt;key&gt;&quot;, &quot;maxPct&quot;: &lt;number&gt; }</code>, the MCP form of the CLI&#39;s repeated <code>--stage-regression-budget &lt;metric&gt;:&lt;pct&gt;</code>. <code>metric</code> is one of <code>executorRunTime</code>, <code>executorCpuTime</code>, <code>memoryBytesSpilled</code>, <code>diskBytesSpilled</code>, <code>shuffleReadBytes</code> and <code>shuffleWriteBytes</code>. The budget fails when any paired stage&#39;s metric grew by more than <code>maxPct</code>% against its baseline stage; growth from a zero baseline always exceeds it. The detail names the worst pairs by <code>pairId</code> (the same id <code>compare_runs</code> returns in <code>stagePairs</code>). Each metric gets its own <code>max-stage-regression</code> result, and a metric can be budgeted once. Requires <code>runIdB</code>/<code>sourceB</code>. It reports <code>inconclusive</code> when no stage pair is eligible or the metric is missing on every eligible pair. <code>inputBytes</code> and <code>outputBytes</code> are workload volume with no regression direction, so the call fails if either is named.</li><li><code>stageQualities</code>: the pair qualities <code>stageRegressionBudgets</code> read, from <code>exact</code>, <code>structural</code> and <code>aligned</code> (default <code>exact</code> and <code>structural</code>). An <code>aligned</code> pair was matched on similarity or position and may compare different work, so it is left out unless named. Requires <code>stageRegressionBudgets</code>. Stages a re-plan left over are not paired and are not read.</li><li><code>failOnIntroduced</code>: fail if the second run introduces any finding in the given impact band (<code>&quot;all&quot;</code> or one of the impact band names). Requires <code>runIdB</code>/<code>sourceB</code>. A band name it doesn&#39;t recognize reports <code>inconclusive</code>.</li></ul><p>A budget whose required evidence is missing (e.g. no <code>runIdB</code>/<code>sourceB</code> for a regression budget, or a run with no trustworthy task-level evidence) reports <code>inconclusive</code>, not a false pass. The tool has no <code>redact</code> input: the budgets read figures and impact bands, never the stage text redaction rewrites, so a redacted comparison gives the same results. Independent of which budgets you pass, an evaluated run with no ApplicationEnd event adds a <code>run-complete</code> result with status <code>inconclusive</code>, the same check that makes the CLI exit <code>3</code>. With two runs, the check applies to the candidate.</p><p>Each result&#39;s <code>name</code> is one of <code>max-runtime</code>, <code>max-spill</code>, <code>max-skew</code>, <code>max-failed-task-rate</code>, <code>min-efficiency</code>, <code>max-regression</code>, <code>max-stage-regression</code>, <code>fail-on-introduced</code> and <code>run-complete</code>. A <code>max-regression</code> or <code>max-stage-regression</code> result also carries <code>metric</code>, the key it checked. The returned <code>runId</code> is the first run&#39;s (the baseline, when two runs are given).</p><p>Example call:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;evaluate_budgets&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;arguments&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: {</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;source&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;path&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;baseline.zstd&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> },</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;sourceB&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;path&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;candidate.zstd&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> },</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;maxRegressionPct&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">10</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;failOnIntroduced&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;critical&quot;</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  }</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">}</span></span></code></pre></div><p>Example response:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;runId&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;aaaa1111-...&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;results&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: [</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">    {</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;max-regression&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;metric&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;wallClock&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;status&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;violation&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;detail&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;Metric </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">\\&quot;</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">wallClock</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">\\&quot;</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> regressed 100.0%, exceeding budget 10%.&quot;</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">    },</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">    {</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;fail-on-introduced&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;status&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;pass&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;detail&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;No introduced findings match </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">\\&quot;</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">critical</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">\\&quot;</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">.&quot;</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">    }</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  ],</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;violated&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">true</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;inconclusive&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">false</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">}</span></span></code></pre></div>`,11)])])}const u=s(n,[["render",o]]);export{k as __pageData,u as default};

package/public/docs/assets/user-guide_mcp-tools_evaluate-budgets.md.BB0nDY3Z.lean.js

@@ -0,0 +1 @@
+import{_ as s,o as i,c as a,a5 as t}from"./chunks/framework.DSg0KOwT.js";const k=JSON.parse('{"title":"evaluate_budgets","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/evaluate-budgets.md","filePath":"user-guide/mcp-tools/evaluate-budgets.md"}'),n={name:"user-guide/mcp-tools/evaluate-budgets.md"};function o(d,e,c,l,r,h){return i(),a("div",null,[...e[0]||(e[0]=[t("",11)])])}const u=s(n,[["render",o]]);export{k as __pageData,u as default};

package/public/docs/assets/user-guide_mcp-tools_get-finding-documentation.md.BtTTHNV6.js

@@ -0,0 +1,14 @@
+import{_ as i,o as n,c as a,a5 as t}from"./chunks/framework.DSg0KOwT.js";const c=JSON.parse('{"title":"get_finding_documentation","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/get-finding-documentation.md","filePath":"user-guide/mcp-tools/get-finding-documentation.md"}'),e={name:"user-guide/mcp-tools/get-finding-documentation.md"};function h(o,s,p,l,k,d){return n(),a("div",null,[...s[0]||(s[0]=[t(`<h1 id="get-finding-documentation" tabindex="-1"><code>get_finding_documentation</code> <a class="header-anchor" href="#get-finding-documentation" aria-label="Permalink to &quot;\`get_finding_documentation\`&quot;">​</a></h1><p>Reference for the <code>get_finding_documentation</code> tool of the SparkForensics MCP server.</p><p>Detection and tuning reference documentation for one finding type, independent of any run: fetch it once per type (not once per finding) and cache it.</p><p>Parameters:</p><ul><li><code>type</code>: <code>string</code> (required): a finding <code>type</code> value, e.g. <code>&quot;skew&quot;</code>, or a detector-level type such as <code>&quot;broadcastSizing&quot;</code>, which returns the documentation of the finding types it emits</li></ul><p>Example call:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{ </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;get_finding_documentation&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;arguments&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;type&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;skew&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> } }</span></span></code></pre></div><p>Example response:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;type&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;skew&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;Task Skew&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;detectionDoc&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: {</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;tag&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;SKEW&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;title&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;Task skew&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;content&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;### \`SKEW\`: Task skew {#skew}</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">\\n\\n</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">A small number of tasks...&quot;</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  },</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;tuningDoc&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: {</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;anchor&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;#bottleneck-skew&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;title&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;Task Skew&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;content&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;# Task Skew</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">\\n\\n</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">...&quot;</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  }</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">}</span></span></code></pre></div><p><code>tuningDoc</code> is <code>null</code> when the finding type has no vendored tuning-doc page. Two types have none: <code>configAudit</code> (its four audited properties each have their own anchor rather than one shared page, so no single anchor resolves) and <code>incompleteRun</code> (no upstream tuning page covers this signal at all). A type whose section lives on a general chapter rather than a bottleneck page (<code>autoscalingChurn</code> on Cluster Tuning, <code>cacheUtilization</code> on Memory Management) returns that whole chapter, with <code>anchor</code> naming the section.</p>`,10)])])}const E=i(e,[["render",h]]);export{c as __pageData,E as default};

package/public/docs/assets/user-guide_mcp-tools_get-finding-documentation.md.BtTTHNV6.lean.js

@@ -0,0 +1 @@
+import{_ as i,o as n,c as a,a5 as t}from"./chunks/framework.DSg0KOwT.js";const c=JSON.parse('{"title":"get_finding_documentation","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/get-finding-documentation.md","filePath":"user-guide/mcp-tools/get-finding-documentation.md"}'),e={name:"user-guide/mcp-tools/get-finding-documentation.md"};function h(o,s,p,l,k,d){return n(),a("div",null,[...s[0]||(s[0]=[t("",10)])])}const E=i(e,[["render",h]]);export{c as __pageData,E as default};

package/public/docs/assets/user-guide_mcp-tools_get-finding-evidence.md.rKyu4wts.js

@@ -0,0 +1,20 @@
+import{_ as i,o as a,c as n,a5 as t}from"./chunks/framework.DSg0KOwT.js";const r=JSON.parse('{"title":"get_finding_evidence","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/get-finding-evidence.md","filePath":"user-guide/mcp-tools/get-finding-evidence.md"}'),e={name:"user-guide/mcp-tools/get-finding-evidence.md"};function h(l,s,p,k,d,E){return a(),n("div",null,[...s[0]||(s[0]=[t(`<h1 id="get-finding-evidence" tabindex="-1"><code>get_finding_evidence</code> <a class="header-anchor" href="#get-finding-evidence" aria-label="Permalink to &quot;\`get_finding_evidence\`&quot;">​</a></h1><p>Reference for the <code>get_finding_evidence</code> tool of the SparkForensics MCP server.</p><p>The finding row for one finding id, the same shape as a <code>diagnose_run</code><code>findings</code> entry. Use it to re-read one finding by id without the rest of the report.</p><p>Parameters (<code>runId</code> and <code>findingId</code> are both required: call <code>diagnose_run</code> first to get a <code>runId</code> and a finding&#39;s <code>id</code>):</p><ul><li><code>runId</code>: <code>string</code></li><li><code>findingId</code>: <code>string</code></li><li><code>redact</code>: <code>boolean</code> (default <code>false</code>), pseudonymizes the app id and any host/IP tokens in the response</li></ul><p>Example call:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{ </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;get_finding_evidence&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;arguments&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;runId&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;b8b2c1a4-...&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;findingId&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;1d473001&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> } }</span></span></code></pre></div><p>Example response:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;runId&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;b8b2c1a4-...&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;finding&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: {</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;id&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;1d473001&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;type&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;incompleteRun&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;Incomplete Run&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;tag&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;INCMP&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;impactBand&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;warning&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;stageId&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">null</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;metric&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;applicationEnd&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;value&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">null</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;valueText&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;missing&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;recommendation&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;This event log never recorded an ApplicationEnd event: the capture stopped before the run finished...&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;detectorVersion&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">1</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;evidence&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: {},</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;remediation&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: [],</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;actionLabel&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;incomplete run&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;impactEstimate&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;basis&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;informational&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;wallClock&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">null</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;estimateMethod&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;none&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;coreTimeMs&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">null</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> }</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  }</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">}</span></span></code></pre></div>`,9)])])}const g=i(e,[["render",h]]);export{r as __pageData,g as default};

package/public/docs/assets/user-guide_mcp-tools_get-finding-evidence.md.rKyu4wts.lean.js

@@ -0,0 +1 @@
+import{_ as i,o as a,c as n,a5 as t}from"./chunks/framework.DSg0KOwT.js";const r=JSON.parse('{"title":"get_finding_evidence","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/get-finding-evidence.md","filePath":"user-guide/mcp-tools/get-finding-evidence.md"}'),e={name:"user-guide/mcp-tools/get-finding-evidence.md"};function h(l,s,p,k,d,E){return a(),n("div",null,[...s[0]||(s[0]=[t("",9)])])}const g=i(e,[["render",h]]);export{r as __pageData,g as default};

package/public/docs/assets/user-guide_mcp-tools_get-reference-doc.md.Yac4tH3u.js

@@ -0,0 +1,5 @@
+import{_ as s,o as t,c as i,a5 as a}from"./chunks/framework.DSg0KOwT.js";const k=JSON.parse('{"title":"get_reference_doc","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/get-reference-doc.md","filePath":"user-guide/mcp-tools/get-reference-doc.md"}'),n={name:"user-guide/mcp-tools/get-reference-doc.md"};function o(r,e,h,l,p,c){return t(),i("div",null,[...e[0]||(e[0]=[a(`<h1 id="get-reference-doc" tabindex="-1"><code>get_reference_doc</code> <a class="header-anchor" href="#get-reference-doc" aria-label="Permalink to &quot;\`get_reference_doc\`&quot;">​</a></h1><p>Reference for the <code>get_reference_doc</code> tool of the SparkForensics MCP server.</p><p>Full tuning-reference chapter or bottleneck markdown for one doc anchor (e.g. <code>&quot;#joins&quot;</code>, <code>&quot;#bottleneck-skew&quot;</code>, <code>&quot;#metric-task-duration&quot;</code>), independent of any run, so a client without browser access can read the same reference material the dashboard links to.</p><p>Parameters:</p><ul><li><code>anchor</code>: <code>string</code> (required): a doc anchor, with or without the leading <code>#</code></li></ul><p>Example call:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{ </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;get_reference_doc&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;arguments&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;anchor&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;#bottleneck-skew&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> } }</span></span></code></pre></div><p>Example response:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;anchor&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;bottleneck-skew&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;title&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;Task Skew&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;content&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;# Task Skew</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">\\n\\n</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">...&quot;</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">}</span></span></code></pre></div><p>The returned <code>anchor</code> is the owning page&#39;s, which can differ from the one you passed: <code>&quot;#metric-task-duration&quot;</code> returns the Metrics Glossary page with <code>&quot;anchor&quot;: &quot;metrics&quot;</code>.</p><p>An anchor that resolves to no known page returns the <code>invalid-anchor</code> error code (see Errors below).</p>`,11)])])}const u=s(n,[["render",o]]);export{k as __pageData,u as default};

package/public/docs/assets/user-guide_mcp-tools_get-reference-doc.md.Yac4tH3u.lean.js

@@ -0,0 +1 @@
+import{_ as s,o as t,c as i,a5 as a}from"./chunks/framework.DSg0KOwT.js";const k=JSON.parse('{"title":"get_reference_doc","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/get-reference-doc.md","filePath":"user-guide/mcp-tools/get-reference-doc.md"}'),n={name:"user-guide/mcp-tools/get-reference-doc.md"};function o(r,e,h,l,p,c){return t(),i("div",null,[...e[0]||(e[0]=[a("",11)])])}const u=s(n,[["render",o]]);export{k as __pageData,u as default};

package/public/docs/assets/user-guide_mcp-tools_get-run-summary.md.DpxZ4ZNp.js

@@ -0,0 +1,15 @@
+import{_ as i,o as a,c as t,a5 as n}from"./chunks/framework.DSg0KOwT.js";const E=JSON.parse('{"title":"get_run_summary","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/get-run-summary.md","filePath":"user-guide/mcp-tools/get-run-summary.md"}'),e={name:"user-guide/mcp-tools/get-run-summary.md"};function h(k,s,l,p,o,r){return a(),t("div",null,[...s[0]||(s[0]=[n(`<h1 id="get-run-summary" tabindex="-1"><code>get_run_summary</code> <a class="header-anchor" href="#get-run-summary" aria-label="Permalink to &quot;\`get_run_summary\`&quot;">​</a></h1><p>Reference for the <code>get_run_summary</code> tool of the SparkForensics MCP server.</p><p>App/stage/job/sql counts, duration, and how the run ended: no findings. <code>failedJobs</code>/<code>totalJobs</code> count only jobs with an end record, and <code>failureReason</code> is the first line of Spark&#39;s own recorded reason when a job failed, the same line the dashboard verdict quotes (<code>failureReasonStageId</code> is the stage it came from, or <code>null</code> when there is no reason or it came from a job&#39;s exception). <code>runShape</code> carries the dashboard&#39;s run-shape figures: <code>wallClockMs</code>, <code>efficiencyPct</code> (the share of the run with a stage running, the Scorecard&#39;s Efficiency), <code>unusedCoreTimePct</code> (the share of executor core time that ran no task, against peak concurrent cores; <code>minEfficiencyPct</code> checks 100 minus this), <code>etlPhasesMs</code> (<code>extract</code>/<code>transform</code>/<code>load</code> summed stage time, so a phase can exceed the run) and <code>peakBusyCores</code> (busy cores at the peak of Core Usage by Locality, averaged over one chart bucket, so it can be fractional). Each is <code>null</code> where the dashboard shows &quot;Not measured&quot; or &quot;Unavailable&quot;.</p><p>Parameters: <code>source</code>, <code>runId</code> and <code>redact</code>, as in <code>diagnose_run</code>.</p><p>Example call:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{ </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;get_run_summary&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;arguments&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;source&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;path&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;app-20260101.zstd&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> } } }</span></span></code></pre></div><p>Example response:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;runId&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;3f9c2b7e-...&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;app&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;id&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;application_0000000000000_0001&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;t&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;sparkVersion&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">null</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> },</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;stageCount&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">0</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;jobCount&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">0</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;sqlExecutionCount&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">0</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;executorCount&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;added&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">0</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;removed&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">0</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> },</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;durationMs&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">100</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;runComplete&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">true</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;failedJobs&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">0</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;totalJobs&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">0</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;failureReason&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">null</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;failureReasonStageId&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">null</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;runShape&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;wallClockMs&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">100</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;efficiencyPct&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">null</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;unusedCoreTimePct&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">null</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;etlPhasesMs&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">null</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;peakBusyCores&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">null</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> }</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">}</span></span></code></pre></div>`,8)])])}const u=i(e,[["render",h]]);export{E as __pageData,u as default};

package/public/docs/assets/user-guide_mcp-tools_get-run-summary.md.DpxZ4ZNp.lean.js

@@ -0,0 +1 @@
+import{_ as i,o as a,c as t,a5 as n}from"./chunks/framework.DSg0KOwT.js";const E=JSON.parse('{"title":"get_run_summary","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/get-run-summary.md","filePath":"user-guide/mcp-tools/get-run-summary.md"}'),e={name:"user-guide/mcp-tools/get-run-summary.md"};function h(k,s,l,p,o,r){return a(),t("div",null,[...s[0]||(s[0]=[n("",8)])])}const u=i(e,[["render",h]]);export{E as __pageData,u as default};

package/public/docs/assets/user-guide_mcp-tools_list-runs.md.DpQPeCM3.js

@@ -0,0 +1,12 @@
+import{_ as i,o as a,c as e,a5 as t}from"./chunks/framework.DSg0KOwT.js";const k=JSON.parse('{"title":"list_runs","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/list-runs.md","filePath":"user-guide/mcp-tools/list-runs.md"}'),n={name:"user-guide/mcp-tools/list-runs.md"};function o(l,s,r,p,h,d){return a(),e("div",null,[...s[0]||(s[0]=[t(`<h1 id="list-runs" tabindex="-1"><code>list_runs</code> <a class="header-anchor" href="#list-runs" aria-label="Permalink to &quot;\`list_runs\`&quot;">​</a></h1><p>Reference for the <code>list_runs</code> tool of the SparkForensics MCP server.</p><p>List candidate Spark event-log runs from a local directory or a Spark History Server, before diagnosing one with the tools above. Local-mode scanning is non-recursive: only the files and rolling-log subdirectories directly inside <code>dir</code> are considered. Runs are listed newest first.</p><p>Parameters:</p><ul><li><code>dir</code> (string, local mode) or <code>shsBaseUrl</code> (string, SHS mode): exactly one of the two. Passing both fails input validation; passing neither returns <code>access-or-upstream-failure</code>.</li><li><code>namePattern</code> (string, optional): case-insensitive substring match against each run&#39;s name.</li><li><code>minDate</code>/<code>maxDate</code> (string, optional): filter by start time. A value that doesn&#39;t parse as a date fails with <code>invalid-date-filter</code>. In History Server mode the dates are also sent to the server, which may reject an unparseable value first (<code>access-or-upstream-failure</code>).</li><li><code>maxResults</code> (number, optional, default 100): caps the number of runs returned; when more candidates matched, <code>truncated</code> is <code>true</code>.</li><li><code>redact</code> (boolean, optional, default <code>false</code>): pseudonymizes every run&#39;s app id, name and <code>source</code> (see the note at the top of this page).</li></ul><p>Example call:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{ </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;list_runs&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;arguments&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;dir&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;/var/log/spark-events&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> } }</span></span></code></pre></div><p>Example response:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;runs&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: [</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">    {</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;appId&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;application_1700000000000_0001&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;MyApp&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;sparkVersion&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;3.5.0&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;startTime&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;2026-01-01T12:00:00.000Z&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;source&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;path&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;/var/log/spark-events/application_1700000000000_0001&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> }</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">    }</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  ],</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;truncated&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">false</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">}</span></span></code></pre></div><p>Each entry&#39;s <code>source</code> is the same shape <code>diagnose_run</code>/<code>get_run_summary</code> accept as <code>source</code>, so a result row can be passed straight into those tools without re-deriving anything, unless <code>redact</code> is on: redaction replaces <code>source.path</code> (or <code>source.appId</code>) with the pseudonym, so a redacted row can&#39;t be loaded. A row can also carry <code>durationMs</code> (History Server mode only) and <code>source.attemptId</code>.</p>`,10)])])}const u=i(n,[["render",o]]);export{k as __pageData,u as default};

package/public/docs/assets/user-guide_mcp-tools_list-runs.md.DpQPeCM3.lean.js

@@ -0,0 +1 @@
+import{_ as i,o as a,c as e,a5 as t}from"./chunks/framework.DSg0KOwT.js";const k=JSON.parse('{"title":"list_runs","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/list-runs.md","filePath":"user-guide/mcp-tools/list-runs.md"}'),n={name:"user-guide/mcp-tools/list-runs.md"};function o(l,s,r,p,h,d){return a(),e("div",null,[...s[0]||(s[0]=[t("",10)])])}const u=i(n,[["render",o]]);export{k as __pageData,u as default};

package/public/docs/assets/user-guide_run-comparison.md.BUALkohi.js

@@ -0,0 +1 @@
+import{_ as t,o as a,c as o,a5 as n}from"./chunks/framework.DSg0KOwT.js";const g=JSON.parse('{"title":"Run comparison mode","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/run-comparison.md","filePath":"user-guide/run-comparison.md"}'),r={name:"user-guide/run-comparison.md"};function s(i,e,d,h,c,l){return a(),o("div",null,[...e[0]||(e[0]=[n('<h1 id="run-comparison-mode" tabindex="-1">Run comparison mode <a class="header-anchor" href="#run-comparison-mode" aria-label="Permalink to &quot;Run comparison mode&quot;">​</a></h1><p>Put two runs side by side to see whether a tuning change helped.</p><h2 id="starting-a-comparison" tabindex="-1">Starting a comparison <a class="header-anchor" href="#starting-a-comparison" aria-label="Permalink to &quot;Starting a comparison&quot;">​</a></h2><p>From the landing page, click <strong>Compare two runs</strong>. Two slots appear: <strong>Baseline</strong> and <strong>Candidate</strong>.</p><ol><li>Load a run into each slot the same way you&#39;d load a single run (drag and drop, <strong>Choose file</strong>, <strong>Choose rolling-log folder</strong>, or a History Server fetch).</li><li>Click <strong>Compare</strong>.</li></ol><p>The runs parse one after the other through the same background worker. When both finish, the comparison view opens.</p><p>Already looking at a run, for example the one before a change? Click <strong>Compare with another run</strong> in the top bar (or the <strong>More options</strong> menu on a phone). The compare view opens with that run as the baseline, so you only load the candidate, and the open run is not parsed again. <strong>Back to the run</strong> returns to its dashboard.</p><h2 id="reading-the-comparison" tabindex="-1">Reading the comparison <a class="header-anchor" href="#reading-the-comparison" aria-label="Permalink to &quot;Reading the comparison&quot;">​</a></h2><p>The page opens with a verdict: whether the candidate finished faster or slower than the baseline, and by how much (a change under 2% reads as &quot;about as long&quot;). When either run had failed jobs, the verdict leads with that instead, for example &quot;The candidate had 2 of 5 jobs fail (baseline: none)&quot; or &quot;The candidate&#39;s only job failed&quot;, and the run-time line follows. When either log has no end-of-run record (the run was killed or the log cut off), the verdict says how much run time each log covers, in a neutral tone, instead of calling the shorter run faster. It then lists which cost metrics got worse or better in the candidate (again ignoring changes under 2%), and which finding categories became more or less frequent. Volume and count metrics (input, output, tasks, executors) are left out, since more or less of them is not better or worse on its own. <strong>See where to start in the candidate</strong> opens the candidate&#39;s dashboard, whose own verdict names the first thing to fix.</p><p>The <strong>Metrics</strong> table covers the whole run: wall-clock duration, memory spill, task skew, failed-task rate, disk spill, GC time, input/output bytes, executor run-time, and task/executor counts, baseline against candidate with the change. <strong>Findings by category</strong> lists which finding types became more or less frequent in the candidate, per impact level, with the count in each run and the affected stages.</p><p>The <strong>Stages compared</strong> table lists every paired stage with its change in run time, CPU time, spill (memory and disk together), input, output and shuffle (read and write together), largest run-time change first. A faster candidate reads green and a slower one red; input and output are workload volume, so they stay neutral. Under each stage name sit the pair&#39;s quality (<code>exact</code>, <code>structural</code> or <code>aligned</code>) and its score from 0 to 1. An <code>aligned</code> pair was matched on similarity or position, so treat its change with care.</p><p>Select a stage&#39;s name to see both runs&#39; figures for it side by side without leaving the comparison: the stage&#39;s name in each run, then duration, run time, GC time, spill, input, output and task counts with the change. <strong>Open stage N in the baseline dashboard</strong> and <strong>Open stage N in the candidate dashboard</strong> open that stage&#39;s detail in the run&#39;s own dashboard, and <strong>← Back to comparison</strong> returns here. Below the table, <strong>Re-planned work</strong> lists queries that ran a different number of stages in the two runs, with the leftover stages and their total run time per side, and <strong>Unmatched stages</strong> lists stages that paired with nothing. Both open in the same side-by-side view. The table shows 25 pairs and reveals more on request.</p><p>The table columns read <strong>Baseline</strong> and <strong>Candidate</strong>; hover a column heading to see the full name of the run.</p><p>A banner above the verdict appears when the match is weak. It reads <code>low</code> when under 90% of executor run time sits in paired stages (or the application names differ), and <code>insufficient</code> when neither run recorded any executor run time, so there is nothing to pair. The banner can be dismissed; every change on the page is still shown.</p><p>How stages are paired, and what the matching confidence means, is in <a href="./run-comparison/how-stages-are-matched.html">How stages are matched</a>.</p><p>For everything else, matching is manual: the <strong>Pinned per-stage deltas</strong> widget lets you pick one stage from the baseline and one from the candidate yourself and pin the pair, then shows their duration, executor run-time, GC time, memory/disk spill, input/output bytes, task count, and failed tasks side by side. Pin as many pairs as you want to check.</p><p>For either run&#39;s full dashboard, click <strong>View baseline dashboard</strong> or <strong>View candidate dashboard</strong>. <strong>← Back to comparison</strong> takes you back.</p><h2 id="comparing-without-the-dashboard" tabindex="-1">Comparing without the dashboard <a class="header-anchor" href="#comparing-without-the-dashboard" aria-label="Permalink to &quot;Comparing without the dashboard&quot;">​</a></h2><p>The same comparison runs headlessly, and opens with the same verdict: the positional run is the candidate, the <code>--baseline</code> run the baseline. The CLI&#39;s <code>--baseline &lt;local file or rolling-log dir&gt;</code> flag adds the comparison. In JSON the run&#39;s report moves under <code>candidate</code> and the comparison sits beside it under <code>comparison</code> (<code>comparison.verdict</code> is the headline); in Markdown the comparison section follows the report. <code>--baseline</code> can also gate a build on the comparison (<code>--max-regression-pct</code>, <code>--stage-regression-budget</code>, <code>--fail-on-introduced</code>; see <a href="./getting-started/ci-and-automation.html#ci-and-automation">Getting started</a>). The MCP server&#39;s <code>compare_runs</code> and <code>evaluate_budgets</code> tools do the same for an AI assistant; see <a href="./mcp-tools.html">MCP tools reference</a>.</p>',19)])])}const p=t(r,[["render",s]]);export{g as __pageData,p as default};

package/public/docs/assets/user-guide_run-comparison.md.BUALkohi.lean.js

@@ -0,0 +1 @@
+import{_ as t,o as a,c as o,a5 as n}from"./chunks/framework.DSg0KOwT.js";const g=JSON.parse('{"title":"Run comparison mode","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/run-comparison.md","filePath":"user-guide/run-comparison.md"}'),r={name:"user-guide/run-comparison.md"};function s(i,e,d,h,c,l){return a(),o("div",null,[...e[0]||(e[0]=[n("",19)])])}const p=t(r,[["render",s]]);export{g as __pageData,p as default};

package/public/docs/assets/user-guide_run-comparison_how-stages-are-matched.md.DSXGu2tA.js

@@ -0,0 +1 @@
+import{_ as t,o as a,c as n,a5 as o}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"How stages are matched","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/run-comparison/how-stages-are-matched.md","filePath":"user-guide/run-comparison/how-stages-are-matched.md"}'),s={name:"user-guide/run-comparison/how-stages-are-matched.md"};function i(r,e,d,c,h,l){return a(),n("div",null,[...e[0]||(e[0]=[o('<h1 id="how-stages-are-matched" tabindex="-1">How stages are matched <a class="header-anchor" href="#how-stages-are-matched" aria-label="Permalink to &quot;How stages are matched&quot;">​</a></h1><p>How SparkForensics pairs stages between a baseline and a candidate, and how sure it is about the pairing.</p><p>Stage-level detail depends on pairing a stage in the baseline with its counterpart in the candidate. A stage ID is no help: AQE&#39;s runtime replanning and a different submission order change it from run to run. So a stage is identified by what it ran, and pairing happens in two levels: first the SQL executions of the two runs line up, then the stages inside each pair of executions.</p><h2 id="text-that-differs-between-runs" tabindex="-1">Text that differs between runs <a class="header-anchor" href="#text-that-differs-between-runs" aria-label="Permalink to &quot;Text that differs between runs&quot;">​</a></h2><p>Before two stages are compared, the text that differs between two runs of the same job is rewritten to a fixed form:</p><ul><li>random per-run staging directories in paths (a 6-8 character segment that mixes cases, or letters with digits: a real directory named <code>output01</code> is rewritten too)</li><li>dates and timestamps, such as the run date in a predicate</li><li><code>IN</code> and <code>INSET</code> lists, which grow with the processing window</li><li>file counts in a file index, such as <code>DeltaLogFileIndex(4 paths)</code></li></ul><p>Numbers, paths and string literals in general are not rewritten: two stages that differ in a filter constant or an input path have different exact keys (see below). Text that is specific to your runs, such as an output directory that includes the run&#39;s variant name, can be rewritten with the CLI&#39;s <code>--normalize-path</code> flag or the MCP <code>compare_runs</code> tool&#39;s <code>normalizePath</code> parameter. Despite the name it is a generic regular expression, not only for paths, and it affects only which stages pair, never the findings.</p><h2 id="level-one-sql-executions" tabindex="-1">Level one: SQL executions <a class="header-anchor" href="#level-one-sql-executions" aria-label="Permalink to &quot;Level one: SQL executions&quot;">​</a></h2><p>Each run&#39;s SQL executions are laid out in submission order and aligned with an order-preserving sequence alignment (a longest-common-subsequence dynamic program). An execution pair scores on three things: the call site (the user-code frames in the stages&#39; stack traces), the description, and the plan structure (operator names and the attribute names the plan mentions). A part one side lacks drops out of the score, which matters for PySpark, where every call site is generic. Only pairs scoring at least 0.55 can match, and of equal alignments the one that keeps the lower execution IDs wins, so the result is deterministic and swapping baseline and candidate mirrors it (except when two crossing alignments tie exactly, where the answer depends on which run is the baseline).</p><p>A loop of identical queries that ran 14 times in one run and 15 in the other pairs 14 iterations and leaves the extra iteration unmatched.</p><p>Stages outside any SQL execution (RDD jobs, for example) form one pseudo-execution per run, paired with each other and matched on stage name and call-site text.</p><p>The alignment needs at least half of the smaller run&#39;s executions to pair (<code>paired / min(baseline executions, candidate executions)</code> is at least 0.5). A candidate that removes work, such as caching that drops repeated queries, still pairs everything it kept. Below that the two runs are not treated as one job and no stage pairs: stages as generic as a <code>count</code> or a <code>collect</code> repeat across unrelated jobs, and only the executions around them tell the jobs apart. The comparison reports the counts and the ratio as <code>executionAlignment</code>.</p><h3 id="cost" tabindex="-1">Cost <a class="header-anchor" href="#cost" aria-label="Permalink to &quot;Cost&quot;">​</a></h3><p>The full alignment compares every baseline execution with every candidate execution. Up to 1,000,000 such pairs (1,000 by 1,000 executions) it runs in full. Beyond that it is restricted to a band around the diagonal, about <code>2 * (100 + |n - m| / 2) + 1</code> candidates per baseline execution, so the work grows with the number of executions, not its square. A comparison that used the band reports <code>executionAlignment.bounded: true</code>. An insertion or deletion run inside the band is found; one that shifts the two runs further apart than the band is not, and the executions it separates stay unmatched.</p><h2 id="level-two-stages-inside-an-execution-pair" tabindex="-1">Level two: stages inside an execution pair <a class="header-anchor" href="#level-two-stages-inside-an-execution-pair" aria-label="Permalink to &quot;Level two: stages inside an execution pair&quot;">​</a></h2><p>Every stage has two keys. The <strong>exact key</strong> is its normalized name plus the fingerprint of the plan nodes its tasks were attributed to (normalized node text). The <strong>structural key</strong> is the shape of those nodes: the operator tree, node names and the sorted attribute names each node mentions, with literals, paths, file counts and ids left out. A grouping set that a program emits in a different column order each time has one structural key.</p><p>Inside an aligned execution pair, stages pair in this order, and a stage pairs at most once:</p><ol><li><strong><code>exact</code></strong>: the same exact key. Stages that share a key pair off in stage-ID order, as many as both sides have. A self-join subtree that ran 3 times in one run and 2 in the other pairs 2.</li><li><strong><code>structural</code></strong>: the same structural key, different text.</li><li><strong><code>aligned</code></strong>: different structure, but the text (stage name and normalized node text, compared as sets of tokens) is at least 60% similar. The best scoring pairs go first.</li><li><strong><code>aligned</code></strong>, by position: stages with no attributed plan nodes pair by position among the stages of the same name inside their execution. These pairs have the score 0.5, since no plan evidence backs them.</li></ol><p><code>score</code> is 1 for an exact pair, the text similarity for a structural or aligned one, and 0.5 for a pair made by position. A caller that wants only pairs it can trust reads <code>quality</code>.</p><h2 id="re-planned-executions" tabindex="-1">Re-planned executions <a class="header-anchor" href="#re-planned-executions" aria-label="Permalink to &quot;Re-planned executions&quot;">​</a></h2><p>A configuration change can make Spark plan an execution differently: a broadcast join takes out an exchange, so the two sides run different numbers of stages. When an aligned execution pair has different stage counts (Delta bookkeeping stages excluded), the stages left unpaired on either side are reported together in <code>replanned</code>, with both execution IDs, the leftover stage IDs per side and the total of each delta metric per side. Stages that did pair keep their pair. When the stage counts are equal, leftover stages are <code>unmatched</code> instead: the same number of stages with no pairing point at changed work, not at a re-plan. The stages outside any SQL execution have no plan to re-plan, so their leftovers are <code>unmatched</code>.</p><p>Stages that only read the Delta transaction log or its checkpoints are table bookkeeping, not query work. They are listed apart from the pairs and counted in neither the pairs, the unmatched stages, the replanned stages nor the coverage.</p><h2 id="runtime-coverage-and-confidence" tabindex="-1">Runtime coverage and confidence <a class="header-anchor" href="#runtime-coverage-and-confidence" aria-label="Permalink to &quot;Runtime coverage and confidence&quot;">​</a></h2><p>The comparison reports the <strong>runtime coverage</strong>: the share of both runs&#39; executor run time that sits in paired or replanned stages, with every task attempt counted (failed and speculative ones too). It is weighted by run time, not stage count, because a few unmatched heavy stages matter more than many matched small ones. The share of stages matched, by count, is reported too (<code>matchedCoverage</code>) and does not decide the confidence.</p><p>The confidence is:</p><ul><li><code>ok</code> when the runs have the same application name and the runtime coverage is at least 90%.</li><li><code>low</code> when the application names differ, or the runtime coverage is under 90%. A warning banner says which, for example &quot;Only 62% of executor run time is in matched stages, so per-stage rows mostly compare unrelated work&quot;. The metric deltas above it still hold: they do not depend on stage pairing.</li><li><code>insufficient</code> when neither run recorded any executor run time, so there is no work to compare. The banner says so.</li></ul><p>Replanned run time counts toward the coverage because the two sides of a replanned group still ran the same execution, and the group reports comparable totals. A group shows how much run time it holds, so a coverage made mostly of replanned work is visible in <code>replanned</code>.</p><p>The <strong>Per-stage task skew</strong> table lists only paired stages, and the page states the share of executor run time they hold. Confidence does not change the exit code of a CLI budget.</p>',28)])])}const g=t(s,[["render",i]]);export{p as __pageData,g as default};

package/public/docs/assets/user-guide_run-comparison_how-stages-are-matched.md.DSXGu2tA.lean.js

@@ -0,0 +1 @@
+import{_ as t,o as a,c as n,a5 as o}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"How stages are matched","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/run-comparison/how-stages-are-matched.md","filePath":"user-guide/run-comparison/how-stages-are-matched.md"}'),s={name:"user-guide/run-comparison/how-stages-are-matched.md"};function i(r,e,d,c,h,l){return a(),n("div",null,[...e[0]||(e[0]=[o("",28)])])}const g=t(s,[["render",i]]);export{p as __pageData,g as default};

package/public/docs/assets/user-guide_understanding-findings.md.C9YRruMT.js

@@ -0,0 +1 @@
+import{_ as a,o as t,c as o,a5 as s}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Understanding findings","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/understanding-findings.md","filePath":"user-guide/understanding-findings.md"}'),r={name:"user-guide/understanding-findings.md"};function i(n,e,d,c,l,h){return t(),o("div",null,[...e[0]||(e[0]=[s('<h1 id="understanding-findings" tabindex="-1">Understanding findings <a class="header-anchor" href="#understanding-findings" aria-label="Permalink to &quot;Understanding findings&quot;">​</a></h1><p>Every flagged problem carries a short ALL-CAPS tag. This page has one entry per tag: what it means, and what to do about it.</p><p>Each tag links to the matching section of the Spark <a href="./../tuning-reference/">tuning reference</a>, opened in an in-app <strong>Reference</strong> panel next to the board. <code>INCMP</code> has no reference section, so its tag opens this page instead. Turn on <a href="./getting-started/reading-the-dashboard.html#advanced-view">Advanced view</a> to also see a finding&#39;s confidence when it is below high, and a page icon beside the tag that opens this page&#39;s entry. Some tags point into another tag&#39;s reference page because the material overlaps: <code>SFAIL</code> shares <code>FAIL</code>&#39;s section; <code>PART</code> is on the <code>SHFL</code> page, <code>SPEC</code> on <code>STRAG</code>, <code>SHAPE</code> on <code>SKEW</code>, <code>SLOW</code> on <code>HOST</code>, and <code>CACHE</code>/<code>LOCAL</code> on <code>UTIL</code>.</p><h2 id="per-stage" tabindex="-1">Per-stage <a class="header-anchor" href="#per-stage" aria-label="Permalink to &quot;Per-stage&quot;">​</a></h2><h3 id="skew" tabindex="-1"><code>SKEW</code>: Task skew <a class="header-anchor" href="#skew" aria-label="Permalink to &quot;`SKEW`: Task skew {#skew}&quot;">​</a></h3><p>A small number of tasks take much longer than their peers in the same stage. The fix depends on what the stage reads, which the finding&#39;s <code>evidence.origin</code> records. A stage that reads a shuffle feeding a join (<code>shuffleJoin</code>) gets AQE skew-join handling (<code>spark.sql.adaptive.skewJoin.enabled</code>), unless the run&#39;s effective conf already has it; otherwise salt the key or repartition on a better key. A stage that reads files with uneven sizes (<code>inputScan</code>) gets compaction of small files or a lower <code>spark.sql.files.maxPartitionBytes</code>. Any other stage (<code>other</code>) gets the salting advice and no conf. Flagged when P95 task time (the longest task, on a stage with fewer than 20 tasks) exceeds 3x the median and the recoverable tail is at least 0.5% of the run. The median is the textbook one: on an even task count, the mean of the two middle values.</p><h3 id="shfl" tabindex="-1"><code>SHFL</code>: Shuffle I/O <a class="header-anchor" href="#shfl" aria-label="Permalink to &quot;`SHFL`: Shuffle I/O {#shfl}&quot;">​</a></h3><p>Tasks move a large amount of intermediate data between stages. Raise <code>spark.sql.shuffle.partitions</code>, or add a broadcast join. The partition-count advice follows the effective conf (<code>evidence.partitions</code>): <code>raise</code> when the property limits the stage, <code>sufficient</code> when the tasks already read a good size each, <code>aqeCoalesced</code> when AQE merged the partitions (lower <code>spark.sql.adaptive.advisoryPartitionSizeInBytes</code>), and <code>ownPartitioning</code> when the property is already high enough (the stage&#39;s own <code>repartition(n)</code> or RDD parallelism limits it). Only flagged on stages that take at least 0.5% of the run.</p><h3 id="spill" tabindex="-1"><code>SPILL</code>: Memory and disk spill <a class="header-anchor" href="#spill" aria-label="Permalink to &quot;`SPILL`: Memory and disk spill {#spill}&quot;">​</a></h3><p>Tasks are writing data out of memory, which slows execution. Two spill patterns get flagged differently: skew spill, where a few heavy tasks spill while most don&#39;t (rebalance partitioning), and volume spill, where most tasks spill because the data genuinely exceeds available memory (add partitions or executor memory). <code>spark.sql.shuffle.partitions</code> is suggested only for a stage that reads a shuffle; <code>evidence.reads</code> says what the stage reads (<code>shuffle</code>, <code>input</code> or <code>other</code>). Only flagged on stages that take at least 0.5% of the run.</p><h3 id="gc" tabindex="-1"><code>GC</code>: Garbage collection pressure <a class="header-anchor" href="#gc" aria-label="Permalink to &quot;`GC`: Garbage collection pressure {#gc}&quot;">​</a></h3><p>Tasks spend more than 10% of executor run time reclaiming memory. Reduce object creation: use primitive types, avoid UDFs, or raise executor memory. A stage with GC below 5% gets an informational note that executor memory may be over-provisioned, only on stages that take at least 0.5% of the run. Both need at least 10 s of executor run time on the stage.</p><h3 id="fail" tabindex="-1"><code>FAIL</code>: Failed tasks <a class="header-anchor" href="#fail" aria-label="Permalink to &quot;`FAIL`: Failed tasks {#fail}&quot;">​</a></h3><p>Tasks fail often enough to affect the stage. Failed tasks point to executor instability or data-driven errors. The finding names the dominant error: the exception class, or the executor loss reason (for example &quot;Container killed by YARN for exceeding memory limits&quot;). It lists up to five distinct failures, each with its message and a short stack excerpt. With redaction on, messages become <code>[redacted]</code> and message lines inside excerpts are dropped, since they can carry file paths and data values; class names, stack frames and the executor loss reason stay (hosts in it are pseudonymized).</p><h3 id="sfail" tabindex="-1"><code>SFAIL</code>: Failed stage <a class="header-anchor" href="#sfail" aria-label="Permalink to &quot;`SFAIL`: Failed stage {#sfail}&quot;">​</a></h3><p>A stage attempt failed outright rather than losing individual tasks within it. Inspect the driver log for the failure reason and the job that triggered it. With redaction on, the failure reason becomes <code>[redacted]</code>, since it can carry file paths and data values.</p><h3 id="strag" tabindex="-1"><code>STRAG</code>: Straggler tasks <a class="header-anchor" href="#strag" aria-label="Permalink to &quot;`STRAG`: Straggler tasks {#strag}&quot;">​</a></h3><p>A few tasks run much slower than the rest of their stage. Rule out a GC pause or a slow shuffle fetch before assuming a hardware issue. If uneven data is the cause, the advice is the one a skew finding gives for the same stage (<code>evidence.origin</code>, see <code>SKEW</code>): AQE skew-join handling for a shuffle feeding a join, file sizes for a scan, salting otherwise. Only flagged on stages that take at least 0.5% of the run.</p><h3 id="spec" tabindex="-1"><code>SPEC</code>: Speculation waste <a class="header-anchor" href="#spec" aria-label="Permalink to &quot;`SPEC`: Speculation waste {#spec}&quot;">​</a></h3><p>Speculative task attempts used a lot of executor time without confirming a genuine straggler. Self-flags a confidence that scales with how far the wasted time sits past the threshold. If task durations are just naturally variable rather than genuine stragglers, tune <code>spark.speculation.multiplier</code>/<code>spark.speculation.quantile</code>.</p><h3 id="retry" tabindex="-1"><code>RETRY</code>: Retry waste <a class="header-anchor" href="#retry" aria-label="Permalink to &quot;`RETRY`: Retry waste {#retry}&quot;">​</a></h3><p>Repeated task attempts ate into execution time even though the stage completed. Investigate executor loss or fetch failures.</p><h3 id="tiny" tabindex="-1"><code>TINY</code>: Tiny tasks <a class="header-anchor" href="#tiny" aria-label="Permalink to &quot;`TINY`: Tiny tasks {#tiny}&quot;">​</a></h3><p>Many very short tasks add scheduling overhead out of proportion to the work each one does. Repartition to fewer, larger tasks; a stage that reads a shuffle (<code>evidence.reads</code> is <code>shuffle</code>) can also lower <code>spark.sql.shuffle.partitions</code>. Only flagged on stages that take at least 0.5% of the run.</p><h3 id="part" tabindex="-1"><code>PART</code>: Partition sizing <a class="header-anchor" href="#part" aria-label="Permalink to &quot;`PART`: Partition sizing {#part}&quot;">​</a></h3><p>Shuffle partitions are too large, too uneven, or too few for the work. A single shuffle partition over 5 GB, for example, will OOM or spill heavily: repartition to break it up before the stage runs.</p><h3 id="slow" tabindex="-1"><code>SLOW</code>: Stage slowness <a class="header-anchor" href="#slow" aria-label="Permalink to &quot;`SLOW`: Stage slowness {#slow}&quot;">​</a></h3><p>A stage ran for 15 minutes or more and no slow host was flagged on it. It can appear alongside other findings on the same stage. On a stage that reads a shuffle, often a partition-count problem: raise parallelism via <code>spark.sql.shuffle.partitions</code> or <code>spark.default.parallelism</code>, or check for a large per-task data volume driving heavy shuffle and spill. On a stage that reads input files and no shuffle, check input file sizes and lower <code>spark.sql.files.maxPartitionBytes</code>. <code>evidence.reads</code> says which case applied (<code>shuffle</code>, <code>input</code> or <code>other</code>).</p><h3 id="shape" tabindex="-1"><code>SHAPE</code>: Stage shape <a class="header-anchor" href="#shape" aria-label="Permalink to &quot;`SHAPE`: Stage shape {#shape}&quot;">​</a></h3><p>The stage has an inefficient task count, output shape (output more than 10× input), or task-to-stage balance: one straggler task running for more than half the stage&#39;s wall-clock time and over 3× the median task, so it alone sets when the stage ends. A too-low task count and a straggler are only flagged on stages that take at least 0.5% of the run.</p><h3 id="host" tabindex="-1"><code>HOST</code>: Slow host <a class="header-anchor" href="#host" aria-label="Permalink to &quot;`HOST`: Slow host {#host}&quot;">​</a></h3><p>One executor is much slower than its peers, or carries most of the stage&#39;s task time or bytes. It may just hold data locality for its tasks or carry one heavy stage, rather than a hardware fault. Enable <code>spark.speculation</code> to relaunch a lagging task automatically. Only flagged on stages that take at least 0.5% of the run.</p><h2 id="app-level" tabindex="-1">App-level <a class="header-anchor" href="#app-level" aria-label="Permalink to &quot;App-level&quot;">​</a></h2><h3 id="cold" tabindex="-1"><code>COLD</code>: Executor cold start <a class="header-anchor" href="#cold" aria-label="Permalink to &quot;`COLD`: Executor cold start {#cold}&quot;">​</a></h3><p>The first stage waited more than 30 s for an executor. Keep a warm pool of executors, or, with dynamic allocation, raise <code>spark.dynamicAllocation.minExecutors</code>/<code>initialExecutors</code> so the app doesn&#39;t scale up from zero. With dynamic allocation off (<code>evidence.dynamicAllocation</code> is <code>off</code>) no executor-count property applies and <code>remediation</code> is empty.</p><h3 id="util" tabindex="-1"><code>UTIL</code>: Low utilization <a class="header-anchor" href="#util" aria-label="Permalink to &quot;`UTIL`: Low utilization {#util}&quot;">​</a></h3><p>Allocated executors sit idle for a large share of the application run. Consider a smaller cluster, or enable dynamic allocation.</p><h3 id="mem" tabindex="-1"><code>MEM</code>: Memory utilization <a class="header-anchor" href="#mem" aria-label="Permalink to &quot;`MEM`: Memory utilization {#mem}&quot;">​</a></h3><p>Executor memory or core capacity may be over- or under-provisioned: more than 50% of available core time ran no task, an executor&#39;s heap peaked above 95% of its allocation, or it stayed below 70%. Some detail here needs <code>spark.eventLog.logStageExecutorMetrics=true</code> on the run being analyzed; without it, per-executor memory usage can&#39;t be broken down. Review <code>spark.executor.memory</code> and executor count if allocated memory sat largely idle over the run. That idle-memory variant self-flags a confidence that scales with how far the estimated waste sits past a 1.5x buffer: it estimates waste from allocated memory-time versus task run time (not measured heap usage). Check it against the Spark UI before resizing anything.</p><h3 id="cache" tabindex="-1"><code>CACHE</code>: Caching opportunity <a class="header-anchor" href="#cache" aria-label="Permalink to &quot;`CACHE`: Caching opportunity {#cache}&quot;">​</a></h3><p>A reusable dataset (re-read via the same SQL relation more than once, or a join/union result recomputed by two or more executions, matched by plan shape) may be worth persisting between stages. Self-flags a confidence that scales with how many executions reuse the same relation: reuse is only inferred, from plan-scan identity across SQL executions, so confirm the reads really do hit the same data before you cache anything.</p><h3 id="cstor" tabindex="-1"><code>CSTOR</code>: Cache storage <a class="header-anchor" href="#cstor" aria-label="Permalink to &quot;`CSTOR`: Cache storage {#cstor}&quot;">​</a></h3><p>A persisted dataset is not fully cached in memory, or is spilling to disk. Raise executor memory, or shrink the cached dataset.</p><p>The cached-partition counts and sizes come from <code>SparkListenerBlockUpdated</code> events, which Spark writes only when <code>spark.eventLog.logBlockUpdates.enabled=true</code>. Since Spark 2.3 the RDD storage figures in stage-submission events are always 0, and they are used only as a fallback. When a Spark 2.3+ run persists RDDs but its log has neither and block-update logging was off, the check reports that the cache could not be checked instead of passing it: turn <code>spark.eventLog.logBlockUpdates.enabled</code> on and rerun to measure eviction and disk spillover.</p><h3 id="local" tabindex="-1"><code>LOCAL</code>: Core usage locality <a class="header-anchor" href="#local" aria-label="Permalink to &quot;`LOCAL`: Core usage locality {#local}&quot;">​</a></h3><p>Tasks run without process- or node-local data placement more often than expected. Check executor/data colocation. Self-flags a confidence that scales with the non-local ratio and sample size: the thresholds are our own noise floor for this metric.</p><h3 id="chrn" tabindex="-1"><code>CHRN</code>: Autoscaling churn <a class="header-anchor" href="#chrn" aria-label="Permalink to &quot;`CHRN`: Autoscaling churn {#chrn}&quot;">​</a></h3><p>Executors are stood up and torn down again before they can do useful work: re-provisioning churn rather than normal scale-down. Raise <code>spark.dynamicAllocation.executorIdleTimeout</code>, or widen the <code>minExecutors</code>/<code>maxExecutors</code> bounds to reduce flapping. With dynamic allocation off (<code>evidence.dynamicAllocation</code> is <code>off</code>) none of those applies: the churn is executor loss or preemption, and <code>remediation</code> is empty. Self-flags a confidence that scales with how far the short-lived-executor share sits past the threshold.</p><h3 id="jobs" tabindex="-1"><code>JOBS</code>: Job failure rate <a class="header-anchor" href="#jobs" aria-label="Permalink to &quot;`JOBS`: Job failure rate {#jobs}&quot;">​</a></h3><p>A large share of completed jobs did not succeed. Inspect the driver log for the failed job(s) and the stage failures that triggered them.</p><h3 id="incmp" tabindex="-1"><code>INCMP</code>: Incomplete run <a class="header-anchor" href="#incmp" aria-label="Permalink to &quot;`INCMP`: Incomplete run {#incmp}&quot;">​</a></h3><p>This event log never recorded an <code>ApplicationEnd</code> event: capture stopped before the run finished (an in-flight job, a rotated-away log, or a cut-short capture). Every other finding and metric on the board reflects only what was captured up to that point, not the full run.</p><h2 id="configuration-scope" tabindex="-1">Configuration scope <a class="header-anchor" href="#configuration-scope" aria-label="Permalink to &quot;Configuration scope&quot;">​</a></h2><h3 id="cfg" tabindex="-1"><code>CFG</code>: Configuration audit <a class="header-anchor" href="#cfg" aria-label="Permalink to &quot;`CFG`: Configuration audit {#cfg}&quot;">​</a></h3><p>Flags configuration settings that may cause reliability or efficiency problems, independent of any one stage&#39;s behavior. Four checks run:</p><ul><li><code>spark.shuffle.service.enabled</code>: flagged when dynamic allocation is on but the external shuffle service is off, since shuffle data won&#39;t survive executor removal.</li><li><code>spark.dynamicAllocation.minExecutors</code>/<code>maxExecutors</code>: with dynamic allocation on, flagged when min exceeds max (reported on <code>minExecutors</code>) or when no max is set.</li><li><code>spark.serializer</code>: flagged when not set to Kryo (the default is the Java serializer); <code>org.apache.spark.serializer.KryoSerializer</code> is faster and produces smaller buffers.</li><li><code>spark.executor.memoryOverhead</code>: flagged when set below max(384 MiB, 10% of executor memory).</li></ul><h2 id="sql-scope" tabindex="-1">SQL scope <a class="header-anchor" href="#sql-scope" aria-label="Permalink to &quot;SQL scope&quot;">​</a></h2><h3 id="plan" tabindex="-1"><code>PLAN</code>: Plan advisor <a class="header-anchor" href="#plan" aria-label="Permalink to &quot;`PLAN`: Plan advisor {#plan}&quot;">​</a></h3><p>Flags patterns in the SQL execution plan worth reviewing. Four checks share this tag:</p><ul><li>Duplicate plan subtree: the same subtree recomputed more than once in the plan. When the repeats have the same shape but different filters, columns or tables, the finding stays informational and claims no time. Only flagged when the repeat&#39;s stages take at least 0.5% of the run.</li><li>Small files: one plan node reads or writes more than 100 files averaging under 3 MB. Compact upstream output, or coalesce before writing.</li><li>Under-broadcast: the smaller side of a Sort Merge Join looks well under the broadcast threshold; consider a <code>broadcast()</code> hint or raising <code>spark.sql.autoBroadcastJoinThreshold</code>. When the effective threshold (logged, else Spark&#39;s 10 MiB) already admits the smaller side (<code>evidence.broadcastThreshold</code> is <code>notLimiting</code>), the threshold is not what stopped the broadcast, so <code>remediation</code> is empty and the advice is a hint or table statistics.</li><li>Over-broadcast: a broadcast exceeds the 1 GB threshold; check for a misapplied broadcast hint or a misconfigured <code>spark.sql.autoBroadcastJoinThreshold</code>. When the effective threshold is below the broadcast or auto-broadcast is disabled (<code>evidence.broadcastThreshold</code> is <code>notLimiting</code> or <code>disabled</code>), a hint forced it: remove the hint, and <code>remediation</code> is empty.</li></ul>',60)])])}const f=a(r,[["render",i]]);export{p as __pageData,f as default};

package/public/docs/assets/user-guide_understanding-findings.md.C9YRruMT.lean.js

@@ -0,0 +1 @@
+import{_ as a,o as t,c as o,a5 as s}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Understanding findings","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/understanding-findings.md","filePath":"user-guide/understanding-findings.md"}'),r={name:"user-guide/understanding-findings.md"};function i(n,e,d,c,l,h){return t(),o("div",null,[...e[0]||(e[0]=[s("",60)])])}const f=a(r,[["render",i]]);export{p as __pageData,f as default};