sparkforensics-server 0.5.2 → 0.8.1

package/public/docs/assets/user-guide_mcp-tools_diagnose-run.md.azCZXkH8.js

@@ -0,0 +1,44 @@
+import{_ as i,o as e,c as a,a5 as t}from"./chunks/framework.DSg0KOwT.js";const k=JSON.parse('{"title":"diagnose_run","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/diagnose-run.md","filePath":"user-guide/mcp-tools/diagnose-run.md"}'),n={name:"user-guide/mcp-tools/diagnose-run.md"};function o(d,s,h,l,p,c){return e(),a("div",null,[...s[0]||(s[0]=[t(`<h1 id="diagnose-run" tabindex="-1"><code>diagnose_run</code> <a class="header-anchor" href="#diagnose-run" aria-label="Permalink to &quot;\`diagnose_run\`&quot;">​</a></h1><p>Reference for the <code>diagnose_run</code> tool of the SparkForensics MCP server.</p><p>Diagnose a Spark run: thresholded findings, led by the same verdict the dashboard opens with. <code>verdict</code> gives a <code>title</code> (&quot;Start with Stage 3&quot;, &quot;1 of 3 jobs failed in this run&quot;), <code>summary</code> sentences, <code>steps</code>, the first three places to look in the dashboard&#39;s order (each with its action, what to try, the potential savings and what that figure counts, the other finding types flagged at the same place, and <code>leadFinding</code>, the row of the finding that leads the step), <code>costSteps</code>, the same places ranked by the measured busy core time a fix removes (only findings that carry one), <code>idle</code>, the core-hours and share of the allocation that ran no task, <code>remainingPlaces</code>, how many more places the full list holds, and <code>copyText</code>, the dashboard&#39;s &quot;Copy next steps&quot; checklist. A <code>summary</code> sentence such as &quot;3 findings in 2 places.&quot; counts the findings worth fixing. When the run also has notes on an incomplete log or on data the log lacked, the sentence says the count leaves them out.</p><p>The response names the run and the build that wrote it: <code>app</code> (the Spark application&#39;s <code>id</code> and <code>attemptId</code>, <code>null</code> when the log has none) and <code>generator</code> (the <code>sparkforensics-mcp</code> package <code>name</code> and <code>version</code>, and the <code>buildId</code> of its analysis core). Two diagnoses of one job are comparable when <code>generator.buildId</code> matches. <code>runId</code> is a handle for the run this server process has cached. It differs per tool by design (<code>compare_runs</code> returns <code>runIdA</code> and <code>runIdB</code>, <code>evaluate_budgets</code> the <code>runId</code> of its first run) and per server process, so match runs by <code>app.id</code> and <code>app.attemptId</code>.</p><p>The answer has two views. <code>compact</code> (the default) is an index: the verdict, <code>recommendations</code>, <code>metrics</code>, <code>writeTargets</code>, <code>effectiveConf</code> and one short row per finding, with no <code>cleanChecks</code> or <code>notRunChecks</code>. A compact row holds <code>id</code>, <code>key</code>, <code>type</code>, <code>tag</code>, <code>name</code>, <code>impactBand</code>, <code>stageId</code>, <code>actionLabel</code>, <code>metric</code>, <code>value</code> (or <code>valueText</code>), <code>impact</code> and <code>impactMeaning</code> when the finding has a savings figure, and <code>tunedThresholds</code> on a tuned server. Pass the row&#39;s <code>id</code> to <a href="./get-finding-evidence.html"><code>get_finding_evidence</code></a> for its remediation text, evidence and estimate. <code>full</code> returns every finding as the CLI report writes it, with <code>cleanChecks</code> and <code>notRunChecks</code>. The <code>view</code> field of the response names the view. On the bundled sample run the compact answer is about 40% smaller than the full one.</p><p>Each finding row has a <code>key</code> that is the same for the same finding in a rerun of the job: the finding&#39;s type, its rule and where it sits. A finding on a stage names the stage by the fingerprint that keys <code>metrics.stages</code>, so a row joins to its stage&#39;s metrics without a lookup. A finding on a SQL execution names it by a digest of its stages&#39; fingerprints, and a configuration finding by the property. The key pairs findings across runs. It is not unique (a detector that fires once per host at one place gives rows that share it) and it says nothing about whether the finding&#39;s value or impact band stayed the same. The rows of <a href="./compare-runs.html"><code>compare_runs</code></a> list the same keys. <code>id</code> stays the handle within one run.</p><p>The response also carries the three machine-readable blocks of the CLI&#39;s JSON report, with the same content: <code>writeTargets</code> (every SQL write with its path or table, see <a href="./../getting-started/ci-and-automation.html#write-targets">Write targets</a>), <code>metrics</code> (run totals and one row per stage, see <a href="./../getting-started/ci-and-automation.html#metrics-block">Metrics block</a>) and <code>effectiveConf</code> (the Spark properties the run started with, <code>null</code> when the log records none, see <a href="./../getting-started/ci-and-automation.html#effective-conf">Effective conf</a>). With <code>redact</code> they are built from the redacted run, as in the CLI. The tool has no counterpart of the CLI&#39;s <code>--conf-keys</code> and <code>--conf-redact-regex</code>: <code>effectiveConf</code> always lists every property, withholding Spark&#39;s default secret pattern and the job&#39;s own <code>spark.redaction.regex</code>.</p><p>Parameters (all optional, but pass either a <code>source</code> to load a fresh run, or a <code>runId</code> for one already loaded in this session; passing neither is an error, and when both are given <code>runId</code> wins):</p><ul><li><code>source</code>: <code>{ path: string }</code> or <code>{ shsBaseUrl: string, appId: string, attemptId?: string }</code>. A History Server <code>appId</code> must have one of Spark&#39;s id forms (<code>application_&lt;digits&gt;_&lt;digits&gt;</code>, <code>local-&lt;digits&gt;</code>, <code>app-...</code>, <code>spark-...</code>, <code>driver-&lt;digits&gt;</code>).</li><li><code>runId</code>: <code>string</code></li><li><code>view</code>: <code>&quot;compact&quot; | &quot;full&quot;</code> (default <code>&quot;compact&quot;</code>), see above</li><li><code>redact</code>: <code>boolean</code> (default <code>false</code>), pseudonymizes the app id and any host/IP tokens in the response, and replaces the app name with the app id&#39;s pseudonym</li><li><code>include</code>: array of <code>&quot;summary&quot; | &quot;evidenceAvailability&quot; | &quot;detectors&quot;</code> (default omitted, i.e. none). Each requested value adds one extra top-level field to the response, on top of the default <code>verdict</code>/<code>findings</code>/<code>recommendations</code>/ <code>runComplete</code>/<code>writeTargets</code>/<code>generator</code>/<code>app</code>/<code>metrics</code>/<code>effectiveConf</code> (and <code>cleanChecks</code>/<code>notRunChecks</code> in the <code>full</code> view): <ul><li><code>summary</code>: app id/name/Spark version, stage/job/SQL-execution counts, a finding count broken down by impact band, the same counts without evidence caveats and the incomplete-run row (<code>actionableFindingCount</code>, <code>actionableImpactBandCounts</code>, what the dashboard counts), <code>outcome</code> (how the run ended), <code>runShape</code> (as in <code>get_run_summary</code>), <code>clean</code>, and <code>tunedThresholds</code> on a tuned server</li><li><code>evidenceAvailability</code>: which event types the log actually contained, so you can tell &quot;this check came back clean&quot; apart from &quot;this check couldn&#39;t run because the log is missing data&quot;</li><li><code>detectors</code>: the full catalog of checks this tool can run: type, version, scope (stage/sql/app/config), current thresholds, and a doc-page anchor This is opt-in because <code>detectors</code> in particular is a large, mostly-static catalog that would bloat a routine &quot;what&#39;s wrong with this run&quot; call.</li></ul></li><li><code>impactBand</code>: array of impact bands (e.g. <code>[&quot;critical&quot;, &quot;warning&quot;]</code>), narrows the <code>findings</code> array to only these bands</li><li><code>type</code>: array of finding <code>type</code> values, narrows <code>findings</code> to only these types</li><li><code>stageId</code>: number, narrows <code>findings</code> to only findings on this stage, including a SQL plan finding whose only stage it is (the dashboard&#39;s stage details rule)</li><li><code>minBand</code>: <code>&quot;critical&quot; | &quot;warning&quot; | &quot;info&quot;</code>, lists only findings at this band or a more severe one</li><li><code>maxFindings</code>: non-negative integer, lists at most this many findings after every other filter, most severe first. <code>findingsOmitted</code> counts the matching findings the limit left out (always present, <code>0</code> when none).</li><li><code>format</code>: <code>&quot;json&quot; | &quot;md&quot;</code> (default <code>&quot;json&quot;</code>), switches <code>content[0].text</code> to a rendered Markdown report instead of JSON. With <code>&quot;md&quot;</code>, <code>structuredContent</code> is a small object instead of the report: <code>runId</code>, <code>runComplete</code>, <code>title</code> (the verdict&#39;s), <code>findingCount</code>, <code>findingsOmitted</code> and <code>recommendationCount</code>. A client that stores <code>structuredContent</code> stores only those.</li></ul><p><code>impactBand</code>/<code>type</code>/<code>stageId</code>/<code>minBand</code>/<code>maxFindings</code> only limit the <code>findings</code> array (and the findings section of the Markdown): <code>verdict</code>, <code>recommendations</code>, <code>cleanChecks</code>, <code>notRunChecks</code>, and the finding counts in <code>summary</code> (when requested via <code>include</code>) always stay computed from the full, unfiltered set, so a narrow filter never hides that other checks passed or other fixes exist. Each verdict step keeps its <code>leadFinding</code> row whatever the limit, so the rows the verdict points at are always in the answer.</p><p>A <code>runId</code> expires. Loaded runs are cached in memory, capped at 8 (LRU-evicted) and expired 15 minutes after their last use (override with <code>SPARKFORENSICS_MCP_CACHE_CAP</code> and <code>SPARKFORENSICS_MCP_CACHE_TTL_MS</code>). Loading the same unchanged <code>source</code> again while it is cached returns the same <code>runId</code>. A <code>runId</code> from an earlier <code>diagnose_run</code> call may not resolve when you pass it to <code>get_finding_evidence</code>, <code>compare_runs</code>, or <code>evaluate_budgets</code>; re-load the run with <code>source</code> if you get <code>run-not-found</code>.</p><p>Example call:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{ </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;diagnose_run&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;arguments&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;source&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;path&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;app-20260101.zstd&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> } } }</span></span></code></pre></div><p>Example response in the <code>compact</code> view (an event log with no <code>ApplicationEnd</code> event; fields trimmed: <code>verdict</code>, <code>recommendations</code>, <code>metrics</code>, <code>writeTargets</code>, <code>effectiveConf</code> and the other findings are left out):</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;runId&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;b8b2c1a4-...&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;view&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;compact&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;generator&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;sparkforensics-mcp&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;version&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;0.9.0&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;buildId&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;9c1f3a7e2b4d&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> },</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;app&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;id&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;application_0000000000000_0001&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;attemptId&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">null</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> },</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;runComplete&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">false</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;findingsOmitted&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">0</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;findings&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: [</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">    {</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;id&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;1d473001&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;key&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;incompleteRun|incompleteRun|run&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;type&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;incompleteRun&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;Incomplete Run&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;tag&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;INCMP&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;impactBand&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;warning&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;stageId&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">null</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;actionLabel&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;incomplete run&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;metric&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;applicationEnd&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;value&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">null</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;valueText&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;missing&quot;</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">    }</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  ]</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">}</span></span></code></pre></div><p>The same finding in the <code>full</code> view (the other fields of the response are left out):</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;findings&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: [</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">    {</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;id&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;1d473001&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;key&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;incompleteRun|incompleteRun|run&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;type&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;incompleteRun&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;Incomplete Run&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;tag&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;INCMP&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;impactBand&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;warning&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;stageId&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">null</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;metric&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;applicationEnd&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;value&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">null</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;valueText&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;missing&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;recommendation&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;This event log never recorded an ApplicationEnd event: the capture stopped before the run finished...&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;detectorVersion&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">1</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;evidence&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: {},</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;remediation&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: [],</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;actionLabel&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;incomplete run&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">      &quot;impactEstimate&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;basis&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;informational&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;wallClock&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">null</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;estimateMethod&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;none&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;coreTimeMs&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">null</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> }</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">    }</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  ]</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">}</span></span></code></pre></div><p>The figure that drove a finding is in the row&#39;s <code>metric</code> and <code>value</code> (or <code>valueText</code>). In the <code>full</code> view <code>evidence</code> holds type-specific extra fields and is empty for several types, such as <code>incompleteRun</code> above and <code>skew</code>.</p>`,18)])])}const u=i(n,[["render",o]]);export{k as __pageData,u as default};

package/public/docs/assets/{user-guide_mcp-tools_diagnose-run.md.BP1vnHA5.lean.js → user-guide_mcp-tools_diagnose-run.md.azCZXkH8.lean.js}

@@ -1 +1 @@
-import{_ as e,o as i,c as a,a5 as t}from"./chunks/framework.DSg0KOwT.js";const k=JSON.parse('{"title":"diagnose_run","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/diagnose-run.md","filePath":"user-guide/mcp-tools/diagnose-run.md"}'),n={name:"user-guide/mcp-tools/diagnose-run.md"};function o(d,s,h,l,c,p){return i(),a("div",null,[...s[0]||(s[0]=[t("",13)])])}const u=e(n,[["render",o]]);export{k as __pageData,u as default};
+import{_ as i,o as e,c as a,a5 as t}from"./chunks/framework.DSg0KOwT.js";const k=JSON.parse('{"title":"diagnose_run","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/diagnose-run.md","filePath":"user-guide/mcp-tools/diagnose-run.md"}'),n={name:"user-guide/mcp-tools/diagnose-run.md"};function o(d,s,h,l,p,c){return e(),a("div",null,[...s[0]||(s[0]=[t("",18)])])}const u=i(n,[["render",o]]);export{k as __pageData,u as default};

package/public/docs/assets/{user-guide_mcp-tools_evaluate-budgets.md.BB0nDY3Z.js → user-guide_mcp-tools_evaluate-budgets.md.dpQpRQ7z.js}

@@ -1,4 +1,4 @@
-import{_ as s,o as i,c as a,a5 as t}from"./chunks/framework.DSg0KOwT.js";const k=JSON.parse('{"title":"evaluate_budgets","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/evaluate-budgets.md","filePath":"user-guide/mcp-tools/evaluate-budgets.md"}'),n={name:"user-guide/mcp-tools/evaluate-budgets.md"};function o(d,e,c,l,r,h){return i(),a("div",null,[...e[0]||(e[0]=[t(`<h1 id="evaluate-budgets" tabindex="-1"><code>evaluate_budgets</code> <a class="header-anchor" href="#evaluate-budgets" aria-label="Permalink to &quot;\`evaluate_budgets\`&quot;">​</a></h1><p>Reference for the <code>evaluate_budgets</code> tool of the SparkForensics MCP server.</p><p>Evaluate a run against pass/fail thresholds: the MCP equivalent of the CLI&#39;s <code>--max-runtime</code>/<code>--max-spill</code>/etc. budget flags, thin-wrapped over the same <code>evaluateBudgets()</code> the <code>sparkforensics-analyze</code> CLI uses.</p><p>Parameters (all optional):</p><ul><li><code>source</code> / <code>runId</code>: identify the run to evaluate (same shape as <code>diagnose_run</code>). When <code>runIdB</code>/<code>sourceB</code> is also given, this run is the regression baseline instead.</li><li><code>maxRuntimeMs</code>, <code>maxSpillGb</code>, <code>maxSkewRatio</code>, <code>maxFailedTaskRatePct</code>, <code>minEfficiencyPct</code>: absolute budgets, each evaluated only if provided. With two runs they apply to the candidate (<code>runIdB</code>/<code>sourceB</code>), the same as the CLI&#39;s <code>--baseline</code> mode. <code>minEfficiencyPct</code> measures busy core time, the share of executor core time that ran tasks (100 minus the dashboard&#39;s Unused core time), not the dashboard&#39;s Efficiency tile. <code>maxFailedTaskRatePct</code> reads the task failure rate off the <code>jobFailureRate</code> finding, so when that finding didn&#39;t fire (under 10% of jobs failed, by default) it passes whatever the task failure rate.</li><li><code>runIdB</code> / <code>sourceB</code>: a candidate run to compare against the first, so regression budgets can be evaluated. Same <code>runId</code>/<code>source</code> shape, resolved the same way. Omit both to skip regression budgets.</li><li><code>maxRegressionPct</code>, <code>regressionMetric</code>: fail if <code>regressionMetric</code> (default <code>wallClock</code>; see <a href="./../getting-started/regression-metric-keys.html#regression-metric-keys">Regression metric keys</a> for the full list) regressed by more than <code>maxRegressionPct</code>% between the first run (baseline) and the second run (candidate). Requires <code>runIdB</code>/<code>sourceB</code>. <code>regressionMetric</code> without <code>maxRegressionPct</code> fails the call. A volume key (<code>inputBytes</code>, <code>outputBytes</code>, <code>taskCount</code>, <code>executorsAdded</code>), which has no regression direction, or an unknown key reports <code>inconclusive</code>.</li><li><code>regressionBudgets</code>: further regression budgets, an array of <code>{ &quot;metric&quot;: &quot;&lt;key&gt;&quot;, &quot;maxPct&quot;: &lt;number&gt; }</code>, the MCP form of the CLI&#39;s repeated <code>--regression-budget &lt;metric&gt;:&lt;pct&gt;</code>. Each is checked like <code>maxRegressionPct</code> and reports its own <code>max-regression</code> result. A metric can be budgeted once across this list and the <code>maxRegressionPct</code>/<code>regressionMetric</code> pair (which counts as one budget); a repeat fails the call. Requires <code>runIdB</code>/<code>sourceB</code>, as <code>maxRegressionPct</code> does.</li><li><code>stageRegressionBudgets</code>: per paired stage budgets, an array of <code>{ &quot;metric&quot;: &quot;&lt;key&gt;&quot;, &quot;maxPct&quot;: &lt;number&gt; }</code>, the MCP form of the CLI&#39;s repeated <code>--stage-regression-budget &lt;metric&gt;:&lt;pct&gt;</code>. <code>metric</code> is one of <code>executorRunTime</code>, <code>executorCpuTime</code>, <code>memoryBytesSpilled</code>, <code>diskBytesSpilled</code>, <code>shuffleReadBytes</code> and <code>shuffleWriteBytes</code>. The budget fails when any paired stage&#39;s metric grew by more than <code>maxPct</code>% against its baseline stage; growth from a zero baseline always exceeds it. The detail names the worst pairs by <code>pairId</code> (the same id <code>compare_runs</code> returns in <code>stagePairs</code>). Each metric gets its own <code>max-stage-regression</code> result, and a metric can be budgeted once. Requires <code>runIdB</code>/<code>sourceB</code>. It reports <code>inconclusive</code> when no stage pair is eligible or the metric is missing on every eligible pair. <code>inputBytes</code> and <code>outputBytes</code> are workload volume with no regression direction, so the call fails if either is named.</li><li><code>stageQualities</code>: the pair qualities <code>stageRegressionBudgets</code> read, from <code>exact</code>, <code>structural</code> and <code>aligned</code> (default <code>exact</code> and <code>structural</code>). An <code>aligned</code> pair was matched on similarity or position and may compare different work, so it is left out unless named. Requires <code>stageRegressionBudgets</code>. Stages a re-plan left over are not paired and are not read.</li><li><code>failOnIntroduced</code>: fail if the second run introduces any finding in the given impact band (<code>&quot;all&quot;</code> or one of the impact band names). Requires <code>runIdB</code>/<code>sourceB</code>. A band name it doesn&#39;t recognize reports <code>inconclusive</code>.</li></ul><p>A budget whose required evidence is missing (e.g. no <code>runIdB</code>/<code>sourceB</code> for a regression budget, or a run with no trustworthy task-level evidence) reports <code>inconclusive</code>, not a false pass. The tool has no <code>redact</code> input: the budgets read figures and impact bands, never the stage text redaction rewrites, so a redacted comparison gives the same results. Independent of which budgets you pass, an evaluated run with no ApplicationEnd event adds a <code>run-complete</code> result with status <code>inconclusive</code>, the same check that makes the CLI exit <code>3</code>. With two runs, the check applies to the candidate.</p><p>Each result&#39;s <code>name</code> is one of <code>max-runtime</code>, <code>max-spill</code>, <code>max-skew</code>, <code>max-failed-task-rate</code>, <code>min-efficiency</code>, <code>max-regression</code>, <code>max-stage-regression</code>, <code>fail-on-introduced</code> and <code>run-complete</code>. A <code>max-regression</code> or <code>max-stage-regression</code> result also carries <code>metric</code>, the key it checked. The returned <code>runId</code> is the first run&#39;s (the baseline, when two runs are given).</p><p>Example call:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{</span></span>
+import{_ as s,o as i,c as a,a5 as t}from"./chunks/framework.DSg0KOwT.js";const u=JSON.parse('{"title":"evaluate_budgets","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/evaluate-budgets.md","filePath":"user-guide/mcp-tools/evaluate-budgets.md"}'),n={name:"user-guide/mcp-tools/evaluate-budgets.md"};function o(d,e,c,l,r,h){return i(),a("div",null,[...e[0]||(e[0]=[t(`<h1 id="evaluate-budgets" tabindex="-1"><code>evaluate_budgets</code> <a class="header-anchor" href="#evaluate-budgets" aria-label="Permalink to &quot;\`evaluate_budgets\`&quot;">​</a></h1><p>Reference for the <code>evaluate_budgets</code> tool of the SparkForensics MCP server.</p><p>Evaluate a run against pass/fail thresholds: the MCP equivalent of the CLI&#39;s <code>--max-runtime</code>/<code>--max-spill</code>/etc. budget flags, thin-wrapped over the same <code>evaluateBudgets()</code> the <code>sparkforensics-analyze</code> CLI uses.</p><p>Parameters (all optional):</p><ul><li><code>source</code> / <code>runId</code>: identify the run to evaluate (same shape as <code>diagnose_run</code>). When <code>runIdB</code>/<code>sourceB</code> is also given, this run is the regression baseline instead.</li><li><code>maxRuntimeMs</code>, <code>maxSpillGb</code>, <code>maxSkewRatio</code>, <code>maxFailedTaskRatePct</code>, <code>minEfficiencyPct</code>: absolute budgets, each evaluated only if provided. With two runs they apply to the candidate (<code>runIdB</code>/<code>sourceB</code>), the same as the CLI&#39;s <code>--baseline</code> mode. <code>minEfficiencyPct</code> measures busy core time, the share of executor core time that ran tasks (100 minus the dashboard&#39;s Unused core time), not the dashboard&#39;s Efficiency tile. <code>maxFailedTaskRatePct</code> reads the task failure rate off the <code>jobFailureRate</code> finding, so when that finding didn&#39;t fire (under 10% of jobs failed, by default) it passes whatever the task failure rate.</li><li><code>runIdB</code> / <code>sourceB</code>: a candidate run to compare against the first, so regression budgets can be evaluated. Same <code>runId</code>/<code>source</code> shape, resolved the same way. Omit both to skip regression budgets.</li><li><code>maxRegressionPct</code>, <code>regressionMetric</code>: fail if <code>regressionMetric</code> (default <code>wallClock</code>; see <a href="./../getting-started/regression-metric-keys.html#regression-metric-keys">Regression metric keys</a> for the full list) regressed by more than <code>maxRegressionPct</code>% between the first run (baseline) and the second run (candidate). Requires <code>runIdB</code>/<code>sourceB</code>. <code>regressionMetric</code> without <code>maxRegressionPct</code> fails the call. A volume key (<code>inputBytes</code>, <code>outputBytes</code>, <code>taskCount</code>, <code>executorsAdded</code>), which has no regression direction, or an unknown key reports <code>inconclusive</code>.</li><li><code>regressionBudgets</code>: further regression budgets, an array of <code>{ &quot;metric&quot;: &quot;&lt;key&gt;&quot;, &quot;maxPct&quot;: &lt;number&gt; }</code>, the MCP form of the CLI&#39;s repeated <code>--regression-budget &lt;metric&gt;:&lt;pct&gt;</code>. Each is checked like <code>maxRegressionPct</code> and reports its own <code>max-regression</code> result. A metric can be budgeted once across this list and the <code>maxRegressionPct</code>/<code>regressionMetric</code> pair (which counts as one budget); a repeat fails the call. Requires <code>runIdB</code>/<code>sourceB</code>, as <code>maxRegressionPct</code> does.</li><li><code>stageRegressionBudgets</code>: per paired stage budgets, an array of <code>{ &quot;metric&quot;: &quot;&lt;key&gt;&quot;, &quot;maxPct&quot;: &lt;number&gt; }</code>, the MCP form of the CLI&#39;s repeated <code>--stage-regression-budget &lt;metric&gt;:&lt;pct&gt;</code>. <code>metric</code> is one of <code>executorRunTime</code>, <code>executorCpuTime</code>, <code>memoryBytesSpilled</code>, <code>diskBytesSpilled</code>, <code>shuffleReadBytes</code> and <code>shuffleWriteBytes</code>. The budget fails when any paired stage&#39;s metric grew by more than <code>maxPct</code>% against its baseline stage; growth from a zero baseline always exceeds it. The detail names the worst pairs by <code>pairId</code> (the same id <code>compare_runs</code> returns in <code>stagePairs</code>). Each metric gets its own <code>max-stage-regression</code> result, and a metric can be budgeted once. Requires <code>runIdB</code>/<code>sourceB</code>. It reports <code>inconclusive</code> when no stage pair is eligible or the metric is missing on every eligible pair. <code>inputBytes</code> and <code>outputBytes</code> are workload volume with no regression direction, so the call fails if either is named.</li><li><code>stageQualities</code>: the pair qualities <code>stageRegressionBudgets</code> read, from <code>exact</code>, <code>structural</code> and <code>aligned</code> (default <code>exact</code> and <code>structural</code>). An <code>aligned</code> pair was matched on similarity or position and may compare different work, so it is left out unless named. Requires <code>stageRegressionBudgets</code>. Stages a re-plan left over are not paired and are not read.</li><li><code>failOnIntroduced</code>: fail if the second run introduces any finding in the given impact band (<code>&quot;all&quot;</code> or one of the impact band names). Requires <code>runIdB</code>/<code>sourceB</code>. A finding the first run also has, with the same type, rule and location, is not introduced, even when its impact band differs, unless its band worsens to exactly the given band: that counts as introduced (never under <code>&quot;all&quot;</code>, and an improvement never does). A band name it doesn&#39;t recognize reports <code>inconclusive</code>.</li></ul><p>A budget whose required evidence is missing (e.g. no <code>runIdB</code>/<code>sourceB</code> for a regression budget, or a run with no trustworthy task-level evidence) reports <code>inconclusive</code>, not a false pass. The tool has no <code>redact</code> input: the budgets read figures and impact bands, never the stage text redaction rewrites, so a redacted comparison gives the same results. Independent of which budgets you pass, an evaluated run with no ApplicationEnd event adds a <code>run-complete</code> result with status <code>inconclusive</code>, the same check that makes the CLI exit <code>3</code>. With two runs, the check applies to the candidate.</p><p>Each result&#39;s <code>name</code> is one of <code>max-runtime</code>, <code>max-spill</code>, <code>max-skew</code>, <code>max-failed-task-rate</code>, <code>min-efficiency</code>, <code>max-regression</code>, <code>max-stage-regression</code>, <code>fail-on-introduced</code> and <code>run-complete</code>. A <code>max-regression</code> or <code>max-stage-regression</code> result also carries <code>metric</code>, the key it checked. The returned <code>runId</code> is the first run&#39;s (the baseline, when two runs are given).</p><p>Example call:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{</span></span>
 <span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;evaluate_budgets&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
 <span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;arguments&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: {</span></span>
 <span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;source&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;path&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;baseline.zstd&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> },</span></span>
@@ -23,4 +23,4 @@ import{_ as s,o as i,c as a,a5 as t}from"./chunks/framework.DSg0KOwT.js";const k
 <span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  ],</span></span>
 <span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;violated&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">true</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
 <span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;inconclusive&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">false</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">}</span></span></code></pre></div>`,11)])])}const u=s(n,[["render",o]]);export{k as __pageData,u as default};
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">}</span></span></code></pre></div>`,11)])])}const k=s(n,[["render",o]]);export{u as __pageData,k as default};

package/public/docs/assets/{user-guide_mcp-tools_evaluate-budgets.md.BB0nDY3Z.lean.js → user-guide_mcp-tools_evaluate-budgets.md.dpQpRQ7z.lean.js}

@@ -1 +1 @@
-import{_ as s,o as i,c as a,a5 as t}from"./chunks/framework.DSg0KOwT.js";const k=JSON.parse('{"title":"evaluate_budgets","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/evaluate-budgets.md","filePath":"user-guide/mcp-tools/evaluate-budgets.md"}'),n={name:"user-guide/mcp-tools/evaluate-budgets.md"};function o(d,e,c,l,r,h){return i(),a("div",null,[...e[0]||(e[0]=[t("",11)])])}const u=s(n,[["render",o]]);export{k as __pageData,u as default};
+import{_ as s,o as i,c as a,a5 as t}from"./chunks/framework.DSg0KOwT.js";const u=JSON.parse('{"title":"evaluate_budgets","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/evaluate-budgets.md","filePath":"user-guide/mcp-tools/evaluate-budgets.md"}'),n={name:"user-guide/mcp-tools/evaluate-budgets.md"};function o(d,e,c,l,r,h){return i(),a("div",null,[...e[0]||(e[0]=[t("",11)])])}const k=s(n,[["render",o]]);export{u as __pageData,k as default};

package/public/docs/assets/{user-guide_mcp-tools_get-finding-evidence.md.rKyu4wts.js → user-guide_mcp-tools_get-finding-evidence.md.CH6FkNLX.js}

@@ -1,7 +1,8 @@
-import{_ as i,o as a,c as n,a5 as t}from"./chunks/framework.DSg0KOwT.js";const r=JSON.parse('{"title":"get_finding_evidence","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/get-finding-evidence.md","filePath":"user-guide/mcp-tools/get-finding-evidence.md"}'),e={name:"user-guide/mcp-tools/get-finding-evidence.md"};function h(l,s,p,k,d,E){return a(),n("div",null,[...s[0]||(s[0]=[t(`<h1 id="get-finding-evidence" tabindex="-1"><code>get_finding_evidence</code> <a class="header-anchor" href="#get-finding-evidence" aria-label="Permalink to &quot;\`get_finding_evidence\`&quot;">​</a></h1><p>Reference for the <code>get_finding_evidence</code> tool of the SparkForensics MCP server.</p><p>The finding row for one finding id, the same shape as a <code>diagnose_run</code><code>findings</code> entry. Use it to re-read one finding by id without the rest of the report.</p><p>Parameters (<code>runId</code> and <code>findingId</code> are both required: call <code>diagnose_run</code> first to get a <code>runId</code> and a finding&#39;s <code>id</code>):</p><ul><li><code>runId</code>: <code>string</code></li><li><code>findingId</code>: <code>string</code></li><li><code>redact</code>: <code>boolean</code> (default <code>false</code>), pseudonymizes the app id and any host/IP tokens in the response</li></ul><p>Example call:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{ </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;get_finding_evidence&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;arguments&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;runId&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;b8b2c1a4-...&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;findingId&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;1d473001&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> } }</span></span></code></pre></div><p>Example response:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{</span></span>
+import{_ as i,o as a,c as n,a5 as t}from"./chunks/framework.DSg0KOwT.js";const r=JSON.parse('{"title":"get_finding_evidence","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/get-finding-evidence.md","filePath":"user-guide/mcp-tools/get-finding-evidence.md"}'),e={name:"user-guide/mcp-tools/get-finding-evidence.md"};function h(l,s,k,p,d,E){return a(),n("div",null,[...s[0]||(s[0]=[t(`<h1 id="get-finding-evidence" tabindex="-1"><code>get_finding_evidence</code> <a class="header-anchor" href="#get-finding-evidence" aria-label="Permalink to &quot;\`get_finding_evidence\`&quot;">​</a></h1><p>Reference for the <code>get_finding_evidence</code> tool of the SparkForensics MCP server.</p><p>The finding row for one finding id, the same shape as a <code>diagnose_run</code><code>findings</code> entry. Use it to re-read one finding by id without the rest of the report.</p><p>Parameters (<code>runId</code> and <code>findingId</code> are both required: call <code>diagnose_run</code> first to get a <code>runId</code> and a finding&#39;s <code>id</code>):</p><ul><li><code>runId</code>: <code>string</code></li><li><code>findingId</code>: <code>string</code></li><li><code>redact</code>: <code>boolean</code> (default <code>false</code>), pseudonymizes the app id and any host/IP tokens in the response</li></ul><p>Example call:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{ </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;get_finding_evidence&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;arguments&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;runId&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;b8b2c1a4-...&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;findingId&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;1d473001&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> } }</span></span></code></pre></div><p>Example response:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{</span></span>
 <span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;runId&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;b8b2c1a4-...&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
 <span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;finding&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: {</span></span>
 <span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;id&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;1d473001&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
+<span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;key&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;incompleteRun|incompleteRun|run&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
 <span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;type&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;incompleteRun&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
 <span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;Incomplete Run&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
 <span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;tag&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;INCMP&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>

package/public/docs/assets/{user-guide_mcp-tools_get-finding-evidence.md.rKyu4wts.lean.js → user-guide_mcp-tools_get-finding-evidence.md.CH6FkNLX.lean.js}

@@ -1 +1 @@
-import{_ as i,o as a,c as n,a5 as t}from"./chunks/framework.DSg0KOwT.js";const r=JSON.parse('{"title":"get_finding_evidence","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/get-finding-evidence.md","filePath":"user-guide/mcp-tools/get-finding-evidence.md"}'),e={name:"user-guide/mcp-tools/get-finding-evidence.md"};function h(l,s,p,k,d,E){return a(),n("div",null,[...s[0]||(s[0]=[t("",9)])])}const g=i(e,[["render",h]]);export{r as __pageData,g as default};
+import{_ as i,o as a,c as n,a5 as t}from"./chunks/framework.DSg0KOwT.js";const r=JSON.parse('{"title":"get_finding_evidence","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/get-finding-evidence.md","filePath":"user-guide/mcp-tools/get-finding-evidence.md"}'),e={name:"user-guide/mcp-tools/get-finding-evidence.md"};function h(l,s,k,p,d,E){return a(),n("div",null,[...s[0]||(s[0]=[t("",9)])])}const g=i(e,[["render",h]]);export{r as __pageData,g as default};

package/public/docs/assets/{user-guide_mcp-tools_get-run-summary.md.DpxZ4ZNp.js → user-guide_mcp-tools_get-run-summary.md.EhaIsele.js}

@@ -1,4 +1,4 @@
-import{_ as i,o as a,c as t,a5 as n}from"./chunks/framework.DSg0KOwT.js";const E=JSON.parse('{"title":"get_run_summary","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/get-run-summary.md","filePath":"user-guide/mcp-tools/get-run-summary.md"}'),e={name:"user-guide/mcp-tools/get-run-summary.md"};function h(k,s,l,p,o,r){return a(),t("div",null,[...s[0]||(s[0]=[n(`<h1 id="get-run-summary" tabindex="-1"><code>get_run_summary</code> <a class="header-anchor" href="#get-run-summary" aria-label="Permalink to &quot;\`get_run_summary\`&quot;">​</a></h1><p>Reference for the <code>get_run_summary</code> tool of the SparkForensics MCP server.</p><p>App/stage/job/sql counts, duration, and how the run ended: no findings. <code>failedJobs</code>/<code>totalJobs</code> count only jobs with an end record, and <code>failureReason</code> is the first line of Spark&#39;s own recorded reason when a job failed, the same line the dashboard verdict quotes (<code>failureReasonStageId</code> is the stage it came from, or <code>null</code> when there is no reason or it came from a job&#39;s exception). <code>runShape</code> carries the dashboard&#39;s run-shape figures: <code>wallClockMs</code>, <code>efficiencyPct</code> (the share of the run with a stage running, the Scorecard&#39;s Efficiency), <code>unusedCoreTimePct</code> (the share of executor core time that ran no task, against peak concurrent cores; <code>minEfficiencyPct</code> checks 100 minus this), <code>etlPhasesMs</code> (<code>extract</code>/<code>transform</code>/<code>load</code> summed stage time, so a phase can exceed the run) and <code>peakBusyCores</code> (busy cores at the peak of Core Usage by Locality, averaged over one chart bucket, so it can be fractional). Each is <code>null</code> where the dashboard shows &quot;Not measured&quot; or &quot;Unavailable&quot;.</p><p>Parameters: <code>source</code>, <code>runId</code> and <code>redact</code>, as in <code>diagnose_run</code>.</p><p>Example call:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{ </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;get_run_summary&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;arguments&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;source&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;path&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;app-20260101.zstd&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> } } }</span></span></code></pre></div><p>Example response:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{</span></span>
+import{_ as i,o as a,c as t,a5 as n}from"./chunks/framework.DSg0KOwT.js";const E=JSON.parse('{"title":"get_run_summary","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/mcp-tools/get-run-summary.md","filePath":"user-guide/mcp-tools/get-run-summary.md"}'),e={name:"user-guide/mcp-tools/get-run-summary.md"};function h(k,s,l,p,o,r){return a(),t("div",null,[...s[0]||(s[0]=[n(`<h1 id="get-run-summary" tabindex="-1"><code>get_run_summary</code> <a class="header-anchor" href="#get-run-summary" aria-label="Permalink to &quot;\`get_run_summary\`&quot;">​</a></h1><p>Reference for the <code>get_run_summary</code> tool of the SparkForensics MCP server.</p><p>App/stage/job/sql counts, duration, and how the run ended: no findings. <code>failedJobs</code>/<code>totalJobs</code> count only jobs with an end record, and <code>failureReason</code> is the first line of Spark&#39;s own recorded reason when a job failed, the same line the dashboard verdict quotes (<code>failureReasonStageId</code> is the stage it came from, or <code>null</code> when there is no reason or it came from a job&#39;s exception). <code>runShape</code> carries the dashboard&#39;s run-shape figures: <code>wallClockMs</code>, <code>efficiencyPct</code> (the share of the run with a stage running, the Scorecard&#39;s Efficiency), <code>unusedCoreTimePct</code> (the share of executor core time that ran no task, against the run&#39;s allocation, cores times the time each executor was alive; <code>minEfficiencyPct</code> checks 100 minus this), <code>etlPhasesMs</code> (<code>extract</code>/<code>transform</code>/<code>load</code> summed stage time, so a phase can exceed the run) and <code>peakBusyCores</code> (busy cores at the peak of Core Usage by Locality, averaged over one chart bucket, so it can be fractional). Each is <code>null</code> where the dashboard shows &quot;Not measured&quot; or &quot;Unavailable&quot;.</p><p>Parameters: <code>source</code>, <code>runId</code> and <code>redact</code>, as in <code>diagnose_run</code>.</p><p>Example call:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{ </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;get_run_summary&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;arguments&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;source&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;path&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;app-20260101.zstd&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> } } }</span></span></code></pre></div><p>Example response:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{</span></span>
 <span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;runId&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;3f9c2b7e-...&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
 <span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;app&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;id&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;application_0000000000000_0001&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;t&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;sparkVersion&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">null</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> },</span></span>
 <span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;stageCount&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">0</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>

package/public/docs/assets/user-guide_run-comparison.md.B_LOkdBf.js

@@ -0,0 +1 @@
+import{_ as t,o as a,c as o,a5 as n}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Run comparison mode","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/run-comparison.md","filePath":"user-guide/run-comparison.md"}'),s={name:"user-guide/run-comparison.md"};function r(i,e,d,h,c,l){return a(),o("div",null,[...e[0]||(e[0]=[n('<h1 id="run-comparison-mode" tabindex="-1">Run comparison mode <a class="header-anchor" href="#run-comparison-mode" aria-label="Permalink to &quot;Run comparison mode&quot;">​</a></h1><p>Put two runs side by side to see whether a tuning change helped.</p><h2 id="starting-a-comparison" tabindex="-1">Starting a comparison <a class="header-anchor" href="#starting-a-comparison" aria-label="Permalink to &quot;Starting a comparison&quot;">​</a></h2><p>From the landing page, click <strong>Compare two runs</strong>. Two slots appear: <strong>Baseline</strong> and <strong>Candidate</strong>.</p><ol><li>Load a run into each slot the same way you&#39;d load a single run (drag and drop, <strong>Choose file</strong>, <strong>Choose rolling-log folder</strong>, or a History Server fetch).</li><li>Click <strong>Compare</strong>.</li></ol><p>The runs parse one after the other through the same background worker. When both finish, the comparison view opens.</p><p>Already looking at a run, for example the one before a change? Click <strong>Compare with another run</strong> in the top bar (or the <strong>More options</strong> menu on a phone). The compare view opens with that run as the baseline, so you only load the candidate, and the open run is not parsed again. <strong>Back to the run</strong> returns to its dashboard.</p><h2 id="reading-the-comparison" tabindex="-1">Reading the comparison <a class="header-anchor" href="#reading-the-comparison" aria-label="Permalink to &quot;Reading the comparison&quot;">​</a></h2><p>The page opens with a verdict: whether the candidate finished faster or slower than the baseline, and by how much (a change under 2% reads as &quot;about as long&quot;). When either run had failed jobs, the verdict leads with that instead, for example &quot;The candidate had 2 of 5 jobs fail (baseline: none)&quot; or &quot;The candidate&#39;s only job failed&quot;, and the run-time line follows. When either log has no end-of-run record (the run was killed or the log cut off), the verdict says how much run time each log covers, in a neutral tone, instead of calling the shorter run faster. It then lists which cost metrics got worse or better in the candidate (again ignoring changes under 2%), and which finding categories became more or less frequent. Volume and count metrics (input, output, tasks, executors) are left out, since more or less of them is not better or worse on its own. <strong>See where to start in the candidate</strong> opens the candidate&#39;s dashboard, whose own verdict names the first thing to fix.</p><p>When the two runs have the same Spark configuration, the same plan and the same output, the verdict reads &quot;Same configuration, plan and output&quot; and says where a time difference came from, for example &quot;Time differs by 8m 36s (49%): executors averaged 57.0 in the baseline and 32.6 in the candidate&quot;. A run that had fewer executors, or less of its allocated core time busy, takes longer without being a regression, and the verdict does not call it one. When the time gap is past noise and no measured factor explains it, the verdict keeps its better or worse tone and says the gap is unexplained. When neither run reports output rows or output bytes, the verdict reads &quot;Same configuration and plan. Output was not compared.&quot; and keeps the tone from the wall clock. When configuration or plan differs and the wall clock moves one way and executor run-time or allocated core-hours move the other, the verdict reads as a mixed result and names both.</p><p>The <strong>What differs between the runs</strong> card sits above the metrics. It lists whether the Spark configuration is the same (and, when it is not, each key that differs with both values), whether every stage pairs, the input read, output rows and output written of each run, the mean and peak executor count, and the share of allocated core time that task time filled. Below the table it states how long the two run windows overlapped and the arithmetic that bridges the wall-clock difference: the wall-clock ratio is the task-time ratio divided by the product of the mean-executor ratio and the task-time share ratio, for runs whose executors have the same core count. Configuration keys the cluster sets per run (the application id, start time and driver address) are never compared, and a key whose value is a secret is withheld from both sides and not compared, as in the run output&#39;s effective configuration; the CLI&#39;s <code>--conf-redact-regex</code> matches are withheld too. Output rows must be equal for the output to read as the same; input and output bytes may differ within the noise band. The output reads as the same only when output rows or output bytes are measured on both sides.</p><p>The <strong>Metrics</strong> table covers the whole run: wall-clock duration, memory spill, task skew, failed-task rate, disk spill, GC time, input/output bytes, executor run-time, and task/executor counts, baseline against candidate with the change. <strong>Findings by category</strong> lists the findings that are new in the candidate, the findings that are gone from it, and the findings whose impact level changed, with the affected stages. A finding is the same finding in both runs when it has the same type, rule and location (a paired stage, a paired SQL execution, an audited setting, or the whole run), whatever its value or impact level. A finding the baseline also has at the same place is never new or gone: when its impact level differs, it is listed under <strong>Impact changed</strong> with both levels. A finding on a stage that pairs with nothing still matches the same finding on an unpaired stage in the other run.</p><p>The <strong>Stages compared</strong> table lists every paired stage with its change in run time, CPU time, spill (memory and disk together), input, output and shuffle (read and write together), largest run-time change first. A faster candidate reads green and a slower one red; input and output are workload volume, so they stay neutral. Under each stage name sit the pair&#39;s quality (<code>exact</code>, <code>structural</code> or <code>aligned</code>) and its score from 0 to 1. An <code>aligned</code> pair was matched on similarity or position, so treat its change with care.</p><p>Select a stage&#39;s name to see both runs&#39; figures for it side by side without leaving the comparison: the stage&#39;s name in each run, then duration, run time, GC time, spill, input, output and task counts with the change. <strong>Open stage N in the baseline dashboard</strong> and <strong>Open stage N in the candidate dashboard</strong> open that stage&#39;s detail in the run&#39;s own dashboard, and <strong>← Back to comparison</strong> returns here. Below the table, <strong>Re-planned work</strong> lists queries that ran a different number of stages in the two runs, with the leftover stages and their total run time per side, and <strong>Unmatched stages</strong> lists stages that paired with nothing. Both open in the same side-by-side view, and list stages by name next to their ID. The table shows 25 pairs and reveals more on request.</p><p>When no stage pairs at all, the card says why in one sentence and shows <strong>Pick a stage pair</strong>: choose a baseline and a candidate stage, then <strong>Compare pair</strong> to open them side by side.</p><p>The table columns read <strong>Baseline</strong> and <strong>Candidate</strong>; hover a column heading to see the full name of the run.</p><p>A banner above the verdict appears when the match is weak. It reads <code>low</code> when under 90% of executor run time sits in paired stages (or the application names differ), and <code>insufficient</code> when neither run recorded any executor run time, so there is nothing to pair. The banner can be dismissed; every change on the page is still shown.</p><p>How stages are paired, and what the matching confidence means, is in <a href="./run-comparison/how-stages-are-matched.html">How stages are matched</a>.</p><p>For everything else, matching is manual: the <strong>Pinned per-stage deltas</strong> widget lets you pick one stage from the baseline and one from the candidate yourself and pin the pair, then shows their duration, executor run-time, GC time, memory/disk spill, input/output bytes, task count, and failed tasks side by side. Pin as many pairs as you want to check.</p><p>For either run&#39;s full dashboard, click <strong>View baseline dashboard</strong> or <strong>View candidate dashboard</strong>. <strong>← Back to comparison</strong> takes you back.</p><h2 id="comparing-without-the-dashboard" tabindex="-1">Comparing without the dashboard <a class="header-anchor" href="#comparing-without-the-dashboard" aria-label="Permalink to &quot;Comparing without the dashboard&quot;">​</a></h2><p>The same comparison runs headlessly, and opens with the same verdict: the positional run is the candidate, the <code>--baseline</code> run the baseline. The CLI&#39;s <code>--baseline &lt;local file or rolling-log dir&gt;</code> flag (or <code>--baseline-shs-base-url</code>/<code>--baseline-app-id</code> for a History Server run) adds the comparison. In JSON the run&#39;s report moves under <code>candidate</code> and the comparison sits beside it under <code>comparison</code> (<code>comparison.verdict</code> is the headline); in Markdown the comparison section follows the report. <code>--baseline</code> can also gate a build on the comparison (<code>--max-regression-pct</code>, <code>--stage-regression-budget</code>, <code>--fail-on-introduced</code>; see <a href="./getting-started/ci-and-automation.html#ci-and-automation">Getting started</a>). The MCP server&#39;s <code>compare_runs</code> and <code>evaluate_budgets</code> tools do the same for an AI assistant; see <a href="./mcp-tools.html">MCP tools reference</a>.</p>',22)])])}const g=t(s,[["render",r]]);export{p as __pageData,g as default};

package/public/docs/assets/user-guide_run-comparison.md.B_LOkdBf.lean.js

@@ -0,0 +1 @@
+import{_ as t,o as a,c as o,a5 as n}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Run comparison mode","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/run-comparison.md","filePath":"user-guide/run-comparison.md"}'),s={name:"user-guide/run-comparison.md"};function r(i,e,d,h,c,l){return a(),o("div",null,[...e[0]||(e[0]=[n("",22)])])}const g=t(s,[["render",r]]);export{p as __pageData,g as default};

package/public/docs/assets/user-guide_understanding-findings.md.C4riKCKl.js

@@ -0,0 +1 @@
+import{_ as t,o as a,c as o,a5 as s}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Understanding findings","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/understanding-findings.md","filePath":"user-guide/understanding-findings.md"}'),i={name:"user-guide/understanding-findings.md"};function r(n,e,d,c,h,l){return a(),o("div",null,[...e[0]||(e[0]=[s('<h1 id="understanding-findings" tabindex="-1">Understanding findings <a class="header-anchor" href="#understanding-findings" aria-label="Permalink to &quot;Understanding findings&quot;">​</a></h1><p>Every flagged problem carries a short ALL-CAPS tag. This page has one entry per tag: what it means, and what to do about it.</p><p>Each tag links to the matching section of the Spark <a href="https://shuffle-works.github.io/spark-tuning-reference/" target="_blank" rel="noreferrer">tuning reference</a>, opened in an in-app <strong>Reference</strong> panel next to the board. <code>INCMP</code> has no reference section, so its tag opens this page instead. Turn on <a href="./getting-started/reading-the-dashboard.html#advanced-view">Advanced view</a> to also see a finding&#39;s confidence when it is below high, and a page icon beside the tag that opens this page&#39;s entry. Some tags point into another tag&#39;s reference page because the material overlaps: <code>SFAIL</code> shares <code>FAIL</code>&#39;s section; <code>PART</code> is on the <code>SHFL</code> page, <code>SPEC</code> on <code>STRAG</code>, <code>SHAPE</code> on <code>SKEW</code>, <code>SLOW</code> on <code>HOST</code>, and <code>CACHE</code>/<code>LOCAL</code> on <code>UTIL</code>.</p><h2 id="per-stage" tabindex="-1">Per-stage <a class="header-anchor" href="#per-stage" aria-label="Permalink to &quot;Per-stage&quot;">​</a></h2><h3 id="skew" tabindex="-1"><code>SKEW</code>: Task skew <a class="header-anchor" href="#skew" aria-label="Permalink to &quot;`SKEW`: Task skew {#skew}&quot;">​</a></h3><p>A small number of tasks take much longer than their peers in the same stage. The fix depends on what the stage reads, which the finding&#39;s <code>evidence.origin</code> records. A stage that reads a shuffle feeding a join (<code>shuffleJoin</code>) gets AQE skew-join handling (<code>spark.sql.adaptive.skewJoin.enabled</code>), unless the run&#39;s effective conf already has it; otherwise salt the key or repartition on a better key. A stage that reads files with uneven sizes (<code>inputScan</code>) gets compaction of small files or a lower <code>spark.sql.files.maxPartitionBytes</code>. Any other stage (<code>other</code>) gets the salting advice and no conf, as a <code>code</code> entry in <code>remediation</code>. Flagged when P95 task time (the longest task, on a stage with fewer than 20 tasks) exceeds 3x the median and the recoverable tail is at least 0.5% of the run. The median is the textbook one: on an even task count, the mean of the two middle values.</p><p>Skew is a statement about data volume, as in Spark&#39;s UI and AQE. Each stage&#39;s slow tasks (over 3x the median task) are compared with the median task&#39;s input plus shuffle-read bytes and records. If run time scaled with that volume, the share of the slow tasks&#39; extra time it accounts for is the data share; at 50% or more the tail is skew (<code>evidence.cause</code> is <code>data</code>, <code>evidence.dataRatio</code> is the median slow task&#39;s volume over the median task&#39;s). A tail that mostly is not data is reported as <code>STRAG</code> with its cause. Ratios are taken only against a median task that read at least 1 MiB or 1,000 records, so a median of a few bytes never prints a huge multiple. When the median task read less than that floor (almost nothing), there is no ratio to take (<code>evidence.dataRatio</code> is absent, and the advice says the median task read almost nothing): GC and fetch wait claim their share of the slow tasks&#39; extra time first, and a slow task that read data (at least 1 MiB or 1,000 records) owns what is left. When the data share is under 50%, the cause is the largest of GC, fetch wait, host and what none of them accounts for. With no data volume to compare, what none of them accounts for is labelled <code>unattributed</code> (no share is reported) instead of <code>unexplained</code>; GC, fetch wait or host still win when larger. A stage with no tail attribution is <code>unattributed</code> too. An <code>unattributed</code> tail is reported as <code>STRAG</code> only, never as <code>SKEW</code>.</p><p>With AQE skew-join handling on, a <code>shuffleJoin</code> stage&#39;s advice says why handling did or did not act on that stage&#39;s join, read from the execution&#39;s final plan and its effective conf. <code>evidence.aqeSkew</code> records the case:</p><table tabindex="0"><thead><tr><th><code>aqeSkew</code></th><th>What the final plan and conf show</th><th>Advice</th></tr></thead><tbody><tr><td><code>split</code></td><td>The join is marked <code>skew=true</code> and a shuffle read says <code>skewed</code></td><td>AQE already split the skewed partitions, so what remains is not join skew: look at GC, a slow host or an expensive key. When the tail is data (<code>evidence.cause</code> is <code>data</code>), the slow tasks still read far more data than the median task, and the advice is the key remedy.</td></tr><tr><td><code>evenReads</code></td><td>One join side shuffles next to nothing, so the task read is the other side&#39;s, and the largest task read is under twice the median</td><td>The slow tail is not partition-size skew: look at GC, a slow host or an expensive key. Never given when <code>evidence.cause</code> is <code>data</code>, which counts input and records as well as shuffle-read bytes.</td></tr><tr><td><code>belowThreshold</code></td><td>The largest task read is under <code>spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes</code> (256 MB by default), which bounds every partition of both sides; or one side&#39;s read is at least twice the median but not far enough over it for <code>spark.sql.adaptive.skewJoin.skewedPartitionFactor</code> (5)</td><td>Lower the threshold or the factor for the query.</td></tr><tr><td><code>planShape</code></td><td>An aggregate, window or other operator sits between the join and its shuffle</td><td>AQE splits only a shuffle that feeds the join directly: salt the key.</td></tr><tr><td><code>userRepartition</code></td><td>The shuffle under the join is a <code>repartition</code> or <code>rebalance</code> in the job&#39;s code</td><td>AQE leaves a shuffle you asked for alone: drop it, or salt the key.</td></tr><tr><td><code>joinType</code></td><td>The join type does not let AQE split the skewed side: neither side of a full outer join, only the left side of a left outer, left semi or left anti join, only the right side of a right outer join</td><td>Join the hot key on its own and union the results, or salt the key. Swapping the sides does not help: the null-supplying or filtering table stays unsplittable.</td></tr><tr><td><code>extraShuffle</code></td><td>An aggregate, window or join above this join needs its partitioning, so a split would add a shuffle</td><td>Set <code>spark.sql.adaptive.forceOptimizeSkewedJoin</code> to <code>true</code> when that shuffle costs less than the tail (Spark 3.3 and later; before that, salt the key).</td></tr><tr><td><code>notSplit</code></td><td>Nothing above explains it</td><td>Salt the key.</td></tr></tbody></table><p>The stage&#39;s join is the one whose plan node ran in that stage; a plan whose nodes carry no stage ids is read as one join only when the stage&#39;s shuffle read matches what the join&#39;s two shuffles wrote. A task reads one partition of each join side, so the stage&#39;s largest task read covers both sides. Spark judges each side&#39;s partitions against that side&#39;s own median, which the log does not give, so the finding calls the reads even or under the factor only when one side shuffles at most 5% of the join&#39;s bytes; otherwise it can only show a partition is under the absolute threshold (a task&#39;s read bounds every partition in it). The finding does not say which side is skewed; for a join type that splits only one side it names both possibilities. A run whose final plan is missing or ties no join to the stage, or whose coalesced read could be many small partitions, keeps the general advice.</p><h3 id="shfl" tabindex="-1"><code>SHFL</code>: Shuffle I/O <a class="header-anchor" href="#shfl" aria-label="Permalink to &quot;`SHFL`: Shuffle I/O {#shfl}&quot;">​</a></h3><p>Tasks move a large amount of intermediate data between stages. Raise <code>spark.sql.shuffle.partitions</code>, or add a broadcast join. The partition-count advice follows the effective conf (<code>evidence.partitions</code>): <code>raise</code> when the property limits the stage, <code>sufficient</code> when the tasks already read a good size each, <code>aqeCoalesced</code> when AQE merged the partitions of a stage in a SQL execution (lower <code>spark.sql.adaptive.advisoryPartitionSizeInBytes</code>), and <code>ownPartitioning</code> when the property is already high enough (the stage&#39;s own <code>repartition(n)</code> or RDD parallelism limits it, as for any stage outside a SQL execution). With AQE coalescing on, <code>spark.sql.adaptive.coalescePartitions.initialPartitionNum</code> replaces the property as the starting count when it is set, and a stage that ran more tasks than that, with a skew split in the final plan or no final plan to rule one out, is not blamed on its own <code>repartition(n)</code>: the advice states the task and configured counts and names <code>repartition(n)</code> only when the plan holds one the job wrote. Only flagged on stages that take at least 0.5% of the run.</p><h3 id="spill" tabindex="-1"><code>SPILL</code>: Memory and disk spill <a class="header-anchor" href="#spill" aria-label="Permalink to &quot;`SPILL`: Memory and disk spill {#spill}&quot;">​</a></h3><p>Tasks are writing data out of memory, which slows execution. Two spill patterns get flagged differently: skew spill, where a few heavy tasks spill while most don&#39;t (rebalance partitioning), and volume spill, where most tasks spill because the data genuinely exceeds available memory (add partitions or executor memory). Partition-count advice is given only for a stage that reads a shuffle; <code>evidence.reads</code> says what the stage reads (<code>shuffle</code>, <code>input</code> or <code>other</code>). It names what sized the stage: <code>spark.sql.shuffle.partitions</code> when the stage ran that many tasks, <code>spark.sql.adaptive.advisoryPartitionSizeInBytes</code> when AQE coalesced them, and the stage&#39;s own <code>repartition(n)</code> or RDD parallelism otherwise. Only flagged on stages that take at least 0.5% of the run.</p><h3 id="gc" tabindex="-1"><code>GC</code>: Garbage collection pressure <a class="header-anchor" href="#gc" aria-label="Permalink to &quot;`GC`: Garbage collection pressure {#gc}&quot;">​</a></h3><p>Tasks spend more than 10% of executor run time reclaiming memory. Reduce object creation: use primitive types, avoid UDFs, or raise executor memory. When the log carries no measured executor heap peaks, or the executor memory is unknown, so <a href="#mem"><code>MEM</code></a> cannot judge the heap, a stage with GC below 5% gets an informational note that executor memory may be over-provisioned, only on stages that take at least 0.5% of the run, and never on a stage that failed or lost tasks. Both need at least 10 s of executor run time on the stage. A stage with advice to raise executor memory (high GC, volume spill, a GC-bound straggler tail) gets no advice to lower it.</p><h3 id="fail" tabindex="-1"><code>FAIL</code>: Failed tasks <a class="header-anchor" href="#fail" aria-label="Permalink to &quot;`FAIL`: Failed tasks {#fail}&quot;">​</a></h3><p>Tasks fail often enough to affect the stage. Failed tasks point to executor instability or data-driven errors. The finding names the dominant error: the exception class, or the executor loss reason (for example &quot;Container killed by YARN for exceeding memory limits&quot;). It lists up to five distinct failures, each with its message and a short stack excerpt. With redaction on, messages become <code>[redacted]</code> and message lines inside excerpts are dropped, since they can carry file paths and data values; class names, stack frames and the executor loss reason stay (hosts in it are pseudonymized).</p><h3 id="sfail" tabindex="-1"><code>SFAIL</code>: Failed stage <a class="header-anchor" href="#sfail" aria-label="Permalink to &quot;`SFAIL`: Failed stage {#sfail}&quot;">​</a></h3><p>A stage attempt failed outright rather than losing individual tasks within it. Inspect the driver log for the failure reason and the job that triggered it. With redaction on, the failure reason becomes <code>[redacted]</code>, since it can carry file paths and data values.</p><h3 id="strag" tabindex="-1"><code>STRAG</code>: Straggler tasks <a class="header-anchor" href="#strag" aria-label="Permalink to &quot;`STRAG`: Straggler tasks {#strag}&quot;">​</a></h3><p>A few tasks run much slower than the rest of their stage for a reason other than reading more data (a data-driven tail is <code>SKEW</code>). <code>evidence.cause</code> names what the slow tasks&#39; extra time went to, taken in this order so no millisecond counts twice: data volume, then GC time over the median task&#39;s, then shuffle fetch wait over the median task&#39;s, then one host that holds most of the slow tasks out of proportion to its share of the stage (<code>evidence.host</code>). What none of them accounts for is <code>unexplained</code> (<code>unattributed</code> with no data volume to compare); an <code>unexplained</code> tail whose slow tasks read at least twice the median task&#39;s data says so rather than &quot;no more data&quot;, and keeps the <code>SKEW</code> data advice when data is its largest share; <code>evidence.cpuPct</code> (the slow tasks&#39; CPU time over their run time) says whether those tasks mostly waited or were busy. With no data volume to compare, what is left is <code>unattributed</code> when it is the largest share, as is a stage with no tail attribution: the advice names no fix, only the slow tasks&#39; input sizes, GC time and hosts to check. A stage is flagged when more than 5% of its tasks run over 4x the median (2.5% when the tail clears 0.5% of the run), a speculative task ran, or the <code>SKEW</code> duration test holds on a tail that is not data. A stage whose tail is data but which <code>SKEW</code> does not flag by duration keeps a <code>STRAG</code> finding with cause <code>data</code>. A finding only the <code>SKEW</code> duration test admits reports that test&#39;s ratio (<code>metric</code> <code>P95/median</code> or <code>max/median</code>, <code>unit</code> <code>ratio</code>) instead of a share. Stages under 0.5% of the run are skipped unless the <code>SKEW</code> duration test holds.</p><h3 id="spec" tabindex="-1"><code>SPEC</code>: Speculation waste <a class="header-anchor" href="#spec" aria-label="Permalink to &quot;`SPEC`: Speculation waste {#spec}&quot;">​</a></h3><p>Speculative task attempts used a lot of executor time without confirming a genuine straggler. Self-flags a confidence that scales with how far the wasted time sits past the threshold. If task durations are just naturally variable rather than genuine stragglers, tune <code>spark.speculation.multiplier</code>/<code>spark.speculation.quantile</code>. The recommendation names the run&#39;s effective values: Spark relaunches a task that runs over the multiplier times the median once the quantile of the stage&#39;s tasks has finished, which is 1.5x and 75% before Spark 4.0 and 3x and 90% from 4.0 unless the job sets them. A speculative-attempt <code>straggler</code> and a <code>slowHost</code> finding word their speculation advice the same way.</p><h3 id="retry" tabindex="-1"><code>RETRY</code>: Retry waste <a class="header-anchor" href="#retry" aria-label="Permalink to &quot;`RETRY`: Retry waste {#retry}&quot;">​</a></h3><p>Repeated task attempts ate into execution time even though the stage completed. Investigate executor loss or fetch failures.</p><h3 id="tiny" tabindex="-1"><code>TINY</code>: Tiny tasks <a class="header-anchor" href="#tiny" aria-label="Permalink to &quot;`TINY`: Tiny tasks {#tiny}&quot;">​</a></h3><p>Many very short tasks add scheduling overhead out of proportion to the work each one does. Repartition to fewer, larger tasks. On a stage that reads a shuffle (<code>evidence.reads</code> is <code>shuffle</code>) the advice names what sized it: lower <code>spark.sql.shuffle.partitions</code> when the stage ran that many tasks; when AQE coalesced them and still kept tiny ones, set <code>spark.sql.adaptive.coalescePartitions.parallelismFirst</code> to <code>false</code> (AQE then targets <code>spark.sql.adaptive.advisoryPartitionSizeInBytes</code> rather than <code>defaultParallelism</code> tasks) and then raise the advisory size, which alone changes nothing while <code>parallelismFirst</code> is on; otherwise lower the <code>repartition(n)</code> or RDD partition count in the code. Only flagged on stages that take at least 0.5% of the run.</p><h3 id="part" tabindex="-1"><code>PART</code>: Partition sizing <a class="header-anchor" href="#part" aria-label="Permalink to &quot;`PART`: Partition sizing {#part}&quot;">​</a></h3><p>Shuffle partitions are too large, too uneven, or too few for the work. A single shuffle partition over 5 GB, for example, will OOM or spill heavily: repartition to break it up before the stage runs. The partition-skew rule words its advice the same way a <code>SKEW</code> finding does, including <code>evidence.aqeSkew</code>.</p><h3 id="slow" tabindex="-1"><code>SLOW</code>: Stage slowness <a class="header-anchor" href="#slow" aria-label="Permalink to &quot;`SLOW`: Stage slowness {#slow}&quot;">​</a></h3><p>A stage ran for 15 minutes or more and no slow host was flagged on it. It can appear alongside other findings on the same stage. On a stage that reads a shuffle, often a partition-count problem: raise <code>spark.sql.shuffle.partitions</code> when the stage ran that many tasks, lower <code>spark.sql.adaptive.advisoryPartitionSizeInBytes</code> when AQE coalesced them, or raise the stage&#39;s own <code>repartition(n)</code> or RDD parallelism; or check for a large per-task data volume driving heavy shuffle and spill. On a stage that reads input files and no shuffle, check input file sizes and lower <code>spark.sql.files.maxPartitionBytes</code>. <code>evidence.reads</code> says which case applied (<code>shuffle</code>, <code>input</code> or <code>other</code>).</p><h3 id="shape" tabindex="-1"><code>SHAPE</code>: Stage shape <a class="header-anchor" href="#shape" aria-label="Permalink to &quot;`SHAPE`: Stage shape {#shape}&quot;">​</a></h3><p>The stage has an inefficient task count, output shape (output more than 10× input), or task-to-stage balance: one straggler task running for more than half the stage&#39;s wall-clock time and over 3× the median task, so it alone sets when the stage ends. A too-low task count and a straggler are only flagged on stages that take at least 0.5% of the run.</p><h3 id="host" tabindex="-1"><code>HOST</code>: Slow host <a class="header-anchor" href="#host" aria-label="Permalink to &quot;`HOST`: Slow host {#host}&quot;">​</a></h3><p>One executor is much slower than its peers, or carries most of the stage&#39;s task time or bytes. It may just hold data locality for its tasks or carry one heavy stage, rather than a hardware fault. Enable <code>spark.speculation</code> to relaunch a lagging task automatically. Only flagged on stages that take at least 0.5% of the run. A task-time grade rests on the largest of N executor means, which a larger executor count reaches by chance more often, so the finding states <code>confidence: low</code> with the executor count and the task count.</p><p>An executor whose tasks ran slower than the median executor&#39;s is flagged only when the gap is larger than the stage&#39;s own spread of task times and the executor&#39;s task count allow. The slowest of 75 executors tops the median by chance far more often than the slowest of 3, so the gap it needs grows with the number of executors. A stage whose per-task timings are missing flags no executor on task time.</p><h2 id="app-level" tabindex="-1">App-level <a class="header-anchor" href="#app-level" aria-label="Permalink to &quot;App-level&quot;">​</a></h2><h3 id="cold" tabindex="-1"><code>COLD</code>: Executor cold start <a class="header-anchor" href="#cold" aria-label="Permalink to &quot;`COLD`: Executor cold start {#cold}&quot;">​</a></h3><p>The first stage waited more than 30 s for an executor. Keep a warm pool of executors, or, with dynamic allocation, raise <code>spark.dynamicAllocation.minExecutors</code>/<code>initialExecutors</code> so the app doesn&#39;t scale up from zero. With dynamic allocation off (<code>evidence.dynamicAllocation</code> is <code>off</code>, which includes a run that never set it, as Spark defaults it off) no executor-count property applies and <code>remediation</code> is empty.</p><h3 id="util" tabindex="-1"><code>UTIL</code>: Low utilization <a class="header-anchor" href="#util" aria-label="Permalink to &quot;`UTIL`: Low utilization {#util}&quot;">​</a></h3><p>Allocated executors sit idle for a large share of the application run: under 60% of the core time the run allocated (cores times the time each executor was alive) ran a task. Consider a smaller cluster. With dynamic allocation off, either lower <code>spark.executor.instances</code> or enable dynamic allocation; the two are alternatives. With it on, lower <code>spark.dynamicAllocation.maxExecutors</code>, and <code>spark.dynamicAllocation.minExecutors</code> too when the run sets it above 0.</p><h3 id="mem" tabindex="-1"><code>MEM</code>: Memory utilization <a class="header-anchor" href="#mem" aria-label="Permalink to &quot;`MEM`: Memory utilization {#mem}&quot;">​</a></h3><p>Executor memory or core capacity may be over- or under-provisioned: more than 50% of allocated core time ran no task, or the busiest executor&#39;s heap peak stayed below 70% of <code>spark.executor.memory</code>. Heap peaks come from the executor metrics Spark 3+ writes on every task end (and from stage executor metrics when <code>spark.eventLog.logStageExecutorMetrics=true</code>). Spark samples them at executor heartbeat, so a peak is a lower bound, and the heap-used metric counts uncollected garbage, so a peak near the limit is not reported as a memory risk. A log with no peaks (every task ended between two heartbeats, so each reports zeros, as in local mode) gets a note that executor memory sizing was not measured; it suggests <code>spark.eventLog.logStageExecutorMetrics=true</code> while that is off. A run-level finding that advises lowering executor memory (a heap under 70%, idle allocated memory-time) keeps its figures but not that advice when any stage asks for more memory, and names those stages; a stage&#39;s low-GC note is dropped on a stage that asks for more. The idle-memory figure counts the executors&#39; alive time, not the peak executor count over the whole run. Review <code>spark.executor.memory</code> and executor count if allocated memory sat largely idle over the run. That idle-memory variant self-flags a confidence that scales with how far the estimated waste sits past a 1.5x buffer: it estimates waste from allocated memory-time versus task run time per executor core (not measured heap usage). Check it against the Spark UI before resizing anything.</p><h3 id="cache" tabindex="-1"><code>CACHE</code>: Caching opportunity <a class="header-anchor" href="#cache" aria-label="Permalink to &quot;`CACHE`: Caching opportunity {#cache}&quot;">​</a></h3><p>A reusable dataset (re-read via the same SQL relation more than once, or a join/union result recomputed by two or more executions, matched by plan shape) may be worth persisting between stages. Self-flags a confidence that scales with how many executions reuse the same relation: reuse is only inferred, from plan-scan identity across SQL executions, so confirm the reads really do hit the same data before you cache anything. The figure is the input those queries scanned and how many queries scanned it, not a time.</p><h3 id="cstor" tabindex="-1"><code>CSTOR</code>: Cache storage <a class="header-anchor" href="#cstor" aria-label="Permalink to &quot;`CSTOR`: Cache storage {#cstor}&quot;">​</a></h3><p>A persisted dataset is not fully cached in memory, or is spilling to disk. Raise executor memory, or shrink the cached dataset.</p><p>The cached-partition counts and sizes come from <code>SparkListenerBlockUpdated</code> events, which Spark writes only when <code>spark.eventLog.logBlockUpdates.enabled=true</code>. Since Spark 2.3 the RDD storage figures in stage-submission events are always 0, and they are used only as a fallback. When a Spark 2.3+ run persists RDDs but its log has neither and block-update logging was off, the check reports that the cache could not be checked instead of passing it: turn <code>spark.eventLog.logBlockUpdates.enabled</code> on and rerun to measure eviction and disk spillover.</p><h3 id="local" tabindex="-1"><code>LOCAL</code>: Core usage locality <a class="header-anchor" href="#local" aria-label="Permalink to &quot;`LOCAL`: Core usage locality {#local}&quot;">​</a></h3><p>Tasks run without process- or node-local data placement more often than expected. Check executor/data colocation. Self-flags a confidence that scales with the non-local ratio and sample size: the thresholds are our own noise floor for this metric.</p><h3 id="chrn" tabindex="-1"><code>CHRN</code>: Autoscaling churn <a class="header-anchor" href="#chrn" aria-label="Permalink to &quot;`CHRN`: Autoscaling churn {#chrn}&quot;">​</a></h3><p>Executors are stood up and torn down again before they can do useful work: re-provisioning churn rather than normal scale-down. Raise <code>spark.dynamicAllocation.executorIdleTimeout</code>, or widen the <code>minExecutors</code>/<code>maxExecutors</code> bounds to reduce flapping. With dynamic allocation off (<code>evidence.dynamicAllocation</code> is <code>off</code>) none of those applies: the churn is executor loss or preemption, and <code>remediation</code> is empty. Self-flags a confidence that scales with how far the short-lived-executor share sits past the threshold.</p><h3 id="jobs" tabindex="-1"><code>JOBS</code>: Job failure rate <a class="header-anchor" href="#jobs" aria-label="Permalink to &quot;`JOBS`: Job failure rate {#jobs}&quot;">​</a></h3><p>A large share of completed jobs did not succeed. Inspect the driver log for the failed job(s) and the stage failures that triggered them.</p><h3 id="incmp" tabindex="-1"><code>INCMP</code>: Incomplete run <a class="header-anchor" href="#incmp" aria-label="Permalink to &quot;`INCMP`: Incomplete run {#incmp}&quot;">​</a></h3><p>This event log never recorded an <code>ApplicationEnd</code> event: capture stopped before the run finished (an in-flight job, a rotated-away log, or a cut-short capture). Every other finding and metric on the board reflects only what was captured up to that point, not the full run.</p><h2 id="configuration-scope" tabindex="-1">Configuration scope <a class="header-anchor" href="#configuration-scope" aria-label="Permalink to &quot;Configuration scope&quot;">​</a></h2><h3 id="cfg" tabindex="-1"><code>CFG</code>: Configuration audit <a class="header-anchor" href="#cfg" aria-label="Permalink to &quot;`CFG`: Configuration audit {#cfg}&quot;">​</a></h3><p>Flags configuration settings that may cause reliability or efficiency problems, independent of any one stage&#39;s behavior. Three checks run:</p><ul><li><code>spark.dynamicAllocation.maxExecutors</code>: with dynamic allocation on, flagged when the effective max is Spark&#39;s unbounded default, so the cluster can grow without a cap.</li><li><code>spark.serializer</code>: flagged when not set to Kryo (the default is the Java serializer), and only on a run with stages outside any SQL execution. DataFrame and SQL shuffles and caches use Spark&#39;s own row format, so the serializer only matters for RDD work. <code>org.apache.spark.serializer.KryoSerializer</code> is faster and produces smaller buffers.</li><li><code>spark.executor.memoryOverhead</code>: flagged when set below the overhead Spark computes by default, max(<code>spark.executor.minMemoryOverhead</code>, executor memory times <code>spark.executor.memoryOverheadFactor</code>). The minimum is 384 MiB and the factor 10% unless the run sets them; each setting counts only on a Spark version that reads it (the factor from 3.3, the minimum from 4.0).</li></ul><p>Two settings are never flagged because Spark rejects them at startup, so no event log carries them: dynamic allocation with neither the external shuffle service, shuffle tracking, shuffle-block decommissioning nor a reliable shuffle storage plugin, and <code>minExecutors</code> above <code>maxExecutors</code>.</p><h2 id="sql-scope" tabindex="-1">SQL scope <a class="header-anchor" href="#sql-scope" aria-label="Permalink to &quot;SQL scope&quot;">​</a></h2><h3 id="plan" tabindex="-1"><code>PLAN</code>: Plan advisor <a class="header-anchor" href="#plan" aria-label="Permalink to &quot;`PLAN`: Plan advisor {#plan}&quot;">​</a></h3><p>Flags patterns in the SQL execution plan worth reviewing. Six checks share this tag:</p><ul><li>Duplicate plan subtree: the same subtree recomputed more than once in the plan. When the repeats have the same shape but different filters, columns or tables, the finding stays informational and claims no time. Only flagged when the repeat&#39;s stages take at least 0.5% of the run.</li><li>Small files: one plan node reads or writes more than 100 files averaging under 3 MB. Compact upstream output, or coalesce before writing.</li><li>Nested loop join: a <code>BroadcastNestedLoopJoin</code> or <code>CartesianProduct</code> whose output has at least 1,000,000 rows (the rows a <code>CartesianProduct</code>&#39;s condition kept, when it has one). A <code>BroadcastNestedLoopJoin</code> must also produce at least 10 times the rows of its larger input; a <code>CartesianProduct</code> is judged on its output alone, because it re-reads each input once per partition of the other side, so the input row counts the executors report are not row counts. Spark plans these when a join has no equi-join key. The finding carries the join condition and is graded by the time of the stages that run the join. The advice is to add an equi-join key, to bucket the range of a range join and join on the bucket, or to confirm that a cross join is intended. It needs the executors&#39; <code>number of output rows</code> metrics, so a join with no reported row counts is not flagged.</li><li>Row-at-a-time Python UDF: a plan runs <code>BatchEvalPython</code>, which pickles every row to a Python worker and back. Flagged when its stages sent at least 64 MiB to the workers (<code>data sent to Python workers</code>) and ran for at least 30 seconds together; the finding reports the bytes sent and returned and the stage time, and is informational. <code>ArrowEvalPython</code> (Arrow-optimized and pandas UDFs) is never flagged. The advice is <code>spark.sql.execution.pythonUDF.arrow.enabled=true</code> (Spark 3.4 and later; already the default from 4.2, where the UDF may have opted out with <code>useArrow=False</code>, take no arguments, or have been created before the property was set) or a pandas UDF. These floors are conservative guesses, not tuned against real workloads. A plan that reports no <code>data sent to Python workers</code> value is skipped.</li><li>Under-broadcast: a side of a Sort Merge Join that its join type can broadcast (<code>evidence.buildSide</code>, sized by the finding&#39;s value) was not broadcast although it is small enough: it is over a small effective threshold (<code>evidence.broadcastThreshold</code> is <code>limits</code>), or automatic broadcast is disabled with <code>-1</code> (<code>disabled</code>); consider a <code>broadcast()</code> hint or setting the threshold property. The size is the shuffle&#39;s <code>data size</code>, the metric adaptive execution compares at runtime. The threshold is <code>spark.sql.adaptive.autoBroadcastJoinThreshold</code> when the plan is adaptive and that property is set, else <code>spark.sql.autoBroadcastJoinThreshold</code> (the query&#39;s own setting, else the logged one, else Spark&#39;s 10 MiB). A full outer join never fires, and a side under 1 MiB or over the over-broadcast limit is skipped. When the effective threshold already admits the side (<code>evidence.broadcastThreshold</code> is <code>notLimiting</code>), the threshold is not what stopped the broadcast, so <code>remediation</code> is empty and the advice is a hint or table statistics (for an adaptive plan, <code>spark.sql.adaptive.nonEmptyPartitionRatioForBroadcastJoin</code>: AQE keeps a side with a low share of non-empty partitions out of a broadcast whatever its size). Both sides need a shuffle <code>data size</code> of their own, so a join over another join&#39;s output is skipped, and so is a join whose only buildable side is larger than the other side, since broadcasting it saves nothing. The build side is always the smaller one and <code>evidence.largerSideBytes</code> is the other side.</li><li>Over-broadcast: a broadcast exceeds the 1 GB threshold; check for a misapplied broadcast hint or a misconfigured threshold property. Under adaptive execution a broadcast planned up front is admitted by <code>spark.sql.autoBroadcastJoinThreshold</code> and one converted at runtime by <code>spark.sql.adaptive.autoBroadcastJoinThreshold</code> (when set), so the finding names whichever of them admits the broadcast. When every applicable threshold is below the broadcast or disabled (<code>evidence.broadcastThreshold</code> is <code>notLimiting</code>, or <code>disabled</code> when all are disabled), a hint forced it: remove the hint, and <code>remediation</code> is empty.</li></ul>',66)])])}const f=t(i,[["render",r]]);export{p as __pageData,f as default};

package/public/docs/assets/user-guide_understanding-findings.md.C4riKCKl.lean.js

@@ -0,0 +1 @@
+import{_ as t,o as a,c as o,a5 as s}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Understanding findings","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/understanding-findings.md","filePath":"user-guide/understanding-findings.md"}'),i={name:"user-guide/understanding-findings.md"};function r(n,e,d,c,h,l){return a(),o("div",null,[...e[0]||(e[0]=[s("",66)])])}const f=t(i,[["render",r]]);export{p as __pageData,f as default};