sparkforensics-server 0.5.2 → 0.6.0

package/public/docs/assets/{contributor-guide_architecture_worker-protocol_cli-and-mcp.md.DC1YqXBp.js → contributor-guide_architecture_worker-protocol_cli-and-mcp.md.BX1XnFbq.js}

@@ -1 +1 @@
-import{_ as o,o as c,c as d,a5 as a}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"CLI and MCP server","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/worker-protocol/cli-and-mcp.md","filePath":"contributor-guide/architecture/worker-protocol/cli-and-mcp.md"}'),t={name:"contributor-guide/architecture/worker-protocol/cli-and-mcp.md"};function s(r,e,n,i,l,h){return c(),d("div",null,[...e[0]||(e[0]=[a('<h1 id="cli-and-mcp-server" tabindex="-1">CLI and MCP server <a class="header-anchor" href="#cli-and-mcp-server" aria-label="Permalink to &quot;CLI and MCP server&quot;">​</a></h1><p>The headless analysis CLI and the MCP server, which run the same parser and detectors outside the browser.</p><h2 id="headless-analysis-cli" tabindex="-1">Headless analysis CLI <a class="header-anchor" href="#headless-analysis-cli" aria-label="Permalink to &quot;Headless analysis CLI&quot;">​</a></h2><p><code>packages/cli/bin/sparkforensics-analyze.mjs</code> runs the same parser + detector contracts outside the browser, for CI. It accepts a single event-log file or a rolling-log directory, drives <code>runParse</code>/<code>runParseFiles</code> (<code>packages/core/src/parser-worker.ts</code>) with a Node-only File-like shim (<code>nodeFileFromPath</code> in <code>packages/core/src/cli/collect-run.ts</code>), and writes the exact <code>buildEvidenceReport</code> JSON schema described above: one format, not a second. The shim reads each requested slice with a positioned <code>readSync</code>, so a local log is never loaded whole and has no 2 GiB size limit; <code>collectRun</code> closes every descriptor it opened once parsing settles, on success and on error. Alternatively, <code>--shs-base-url &lt;url&gt; --app-id &lt;id&gt; [--attempt-id &lt;id&gt;]</code> fetches the run from a Spark History Server instead (mutually exclusive with the positional file/directory argument), calling <code>resolveFromShs</code> (<code>packages/core/src/shs-load.ts</code>, shared with the MCP server&#39;s SHS source path below) directly; no dependency on the <code>packages/server</code> package. The three flags are checked first with <code>validateShsRequest</code> (<code>packages/core/src/shs-request.js</code>), and a malformed one is a usage error (exit <code>2</code>) before any fetch. A failed SHS fetch reports its message to <code>stderr</code> and exits <code>4</code>, same as a candidate file that can&#39;t be parsed.</p><p>Regression budgets beyond the <code>--max-regression-pct</code> pair come from repeated <code>--regression-budget &lt;metric&gt;:&lt;pct&gt;</code> flags and a <code>--budgets</code> file, parsed in <code>packages/core/src/cli/regression-budgets.ts</code> and passed to <code>evaluateBudgets()</code> as <code>regressionBudgets</code>; the legacy pair is one more budget, and a metric budgeted twice is a usage error. Per-stage budgets (<code>--stage-regression-budget</code>, <code>--stage-quality</code>; <code>stageRegressionBudgets</code> and <code>stageQualities</code> on MCP) read the aligner&#39;s <code>stagePairs</code> through <code>checkStageRegression</code> in <code>cli/budgets.ts</code> and never match stages themselves; <code>parseStageRegressionBudgetFlag</code> refuses the neutral <code>inputBytes</code> and <code>outputBytes</code>. With two or more positional candidates (or <code>--format ndjson</code>), the CLI&#39;s <code>runMultiLog</code> parses and analyzes the baseline once, then evaluates each candidate in turn and writes one NDJSON line per candidate (<code>log</code>, <code>status</code>, <code>exitCode</code>, <code>error</code>, <code>budgets</code>, <code>candidate</code>, <code>comparison</code>). A candidate that can&#39;t be read or parsed gets a <code>status: &quot;error&quot;</code> line with <code>exitCode</code> <code>4</code> (<code>6</code> for an internal failure on that candidate); the exit code is the worst line (<code>6</code> &gt; <code>5</code> &gt; <code>4</code> &gt; <code>1</code> &gt; <code>3</code> &gt; <code>0</code>, <code>EXIT_SEVERITY</code>). A baseline that can&#39;t be read exits <code>5</code> with no lines. Under <code>--redact</code>, <code>log</code> and the <code>stderr</code> prefixes are <code>candidate-&lt;n&gt;</code> (1-based argument position) and an error line&#39;s message is generic, so no candidate path, which usually carries the app id, reaches the output. In both modes <code>--redact</code> also swaps the <code>stderr</code> text of a baseline or candidate failure (SHS fetch included) for <code>redactedFailure</code>&#39;s role-only message.</p><p>Optional CLI-flag budgets (<code>--max-runtime &lt;ms&gt;</code>, <code>--max-spill &lt;gb&gt;</code>, <code>--max-skew &lt;ratio&gt;</code>, <code>--max-failed-task-rate &lt;pct&gt;</code>, <code>--min-efficiency &lt;pct&gt;</code>) are evaluated in <code>packages/core/src/cli/budgets.ts</code> against the existing finding catalog (<code>analyze()</code>) and <code>computeEfficiencyModel</code>; there is no second rule engine. <code>--min-efficiency</code> compares <code>100 - wastagePct</code> (busy core time, the complement of the dashboard&#39;s Unused core time) and says so in its detail (&quot;Busy core time 26% below budget 90%.&quot;); it is not the Scorecard&#39;s Efficiency tile (<code>stagesActive / total</code>). A budget whose required evidence is missing (e.g. the run never emitted <code>ApplicationEnd</code>, or has no usable per-task <code>runAggregates</code>) is reported as inconclusive (<code>stderr</code> warning) rather than silently passing, and gets its own exit code distinct from both pass and violation. Exit codes: <code>0</code> pass, <code>1</code> a configured budget was violated, <code>2</code> a usage error (unknown or value-less flag, unknown <code>--regression-metric</code> key, an unreadable or invalid <code>--thresholds</code> or <code>--budgets</code> file), <code>3</code> no violations but at least one budget was inconclusive, <code>4</code> the candidate could not be read or parsed (or its SHS fetch failed), <code>5</code> the <code>--baseline</code> could not be read or parsed, <code>6</code> an internal error (an unhandled failure, a failed <code>--export-html</code> export). <code>main</code> catches anything unhandled and exits <code>6</code> rather than letting Node exit <code>1</code>, which would read as a violation. When several apply the worst wins: <code>6</code>, <code>5</code>, <code>4</code>, <code>1</code>, <code>3</code>, <code>0</code>; a violation always wins over an inconclusive result in the same run (exit <code>1</code>, not <code>3</code>). <code>evaluateBudgets()</code> also always adds an inconclusive <code>run-complete</code> result when the catalog has an <code>incompleteRun</code> finding, so a run with no <code>ApplicationEnd</code> exits <code>3</code> even with no budget flags. With <code>--baseline</code>, the absolute budgets and this check apply to the candidate run; the MCP <code>evaluate_budgets</code> tool below uses the same function with the same roles.</p><h2 id="mcp-server" tabindex="-1">MCP server <a class="header-anchor" href="#mcp-server" aria-label="Permalink to &quot;MCP server&quot;">​</a></h2><p><code>packages/core/src/mcp-server-factory.ts</code>&#39;s <code>createMcpServer()</code> registers 8 tools: <code>list_runs</code> (candidate runs in a local directory or on a Spark History Server, to pick one before diagnosing it), <code>diagnose_run</code> (thresholded findings + remediation text), <code>get_run_summary</code> (app/stage/job/sql counts and duration, no findings), <code>compare_runs</code> (the comparison verdict from <code>comparisonVerdict</code> in <code>packages/core/src/comparison-verdict.ts</code>, the dashboard comparison page&#39;s own headline, plus categorized findings delta + metric deltas between two runs; <code>CompareRunsResult.jobOutcomes</code> carries each run&#39;s failed jobs and incomplete flag for it), <code>evaluate_budgets</code> (pass/fail budget thresholds against one run, optionally with a second run for regression/fail-on-introduced budgets: the MCP side of the CLI&#39;s <code>evaluateBudgets()</code> gating), <code>get_finding_evidence</code> (raw evidence bundle for one finding, for drill-down after <code>diagnose_run</code>), <code>get_finding_documentation</code> (detection/tuning reference docs for one finding type, independent of any run), and <code>get_reference_doc</code> (a full tuning-reference chapter or bottleneck page by doc anchor, e.g. <code>#joins</code>). None re-implement detector logic: <code>list_runs</code> lists candidates (<code>packages/core/src/list-runs.ts</code>), and the rest repackage <code>analyze</code>/<code>buildEvidenceReport</code>/<code>compareRuns</code>/<code>captureSnapshot</code>/<code>evaluateBudgets</code> from <code>packages/core/src/mcp-tools.ts</code>, which resolves a <code>source</code> (event-log <code>path</code>, or an SHS <code>shsBaseUrl</code>/<code>appId</code>/<code>attemptId</code> triple) into a cached <code>AppModel</code>. The SHS source path calls <code>resolveFromShs</code> (<code>packages/core/src/shs-load.ts</code>, also used directly by the CLI&#39;s <code>--shs-base-url</code> mode above), which reuses two extractions shared with the browser ingestion flow: <code>fetchShsEventLog</code> (<code>packages/core/src/proxy.js</code>, the same upstream fetch the local server&#39;s <code>/shs-proxy</code> route uses) fetches the zip, and <code>decodeShsArchive</code> (<code>packages/core/src/shs-fetch.ts</code>, re-exported from <code>parser-worker.ts</code>&#39;s barrel) streams it through the parser as worker messages, which <code>collectViaDispatch</code> assembles into an <code>AppModel</code>. <code>runParseFromUrl</code> calls the same <code>decodeShsArchive</code> after fetching through the proxy.</p><p>Two transports connect to that one factory: <code>packages/mcp/bin/sparkforensics-mcp.mjs</code> (stdio, for local MCP clients) and <code>packages/server/index.js</code>&#39;s <code>/mcp</code> route (streamable HTTP, for the local server). The run cache (<code>packages/core/src/mcp-tools.ts</code>) is a module-level LRU (cap 8 and 15-minute idle TTL by default, overridable via <code>SPARKFORENSICS_MCP_CACHE_CAP</code> and <code>SPARKFORENSICS_MCP_CACHE_TTL_MS</code>, lazily swept on access) keyed by resolved source (path plus mtime, ctime and size, or SHS baseUrl+appId+attemptId), so a client mints a <code>runId</code> once via <code>resolveOrCreateRun</code> and reuses it across subsequent tool calls instead of re-parsing.</p><h3 id="one-output-builder-for-the-cli-and-mcp" tabindex="-1">One output builder for the CLI and MCP <a class="header-anchor" href="#one-output-builder-for-the-cli-and-mcp" aria-label="Permalink to &quot;One output builder for the CLI and MCP&quot;">​</a></h3><p>Neither surface lists the fields of a run or a comparison by hand. The CLI&#39;s JSON and the MCP tools call the same core functions:</p><ul><li><code>buildComparisonOutput()</code> in <code>packages/core/src/comparison-output.ts</code> diffs the two runs, applies <code>--redact</code>/<code>redact</code> itself (so no surface can budget or report an unredacted comparison by skipping it), and returns the redacted <code>CompareRunsResult</code> for the budget checks and the Markdown renderer, plus <code>comparisonOutput()</code>&#39;s projection: <code>verdict</code>, <code>confidence</code>, <code>reason</code>, <code>matchedCoverage</code>, <code>runtimeCoverage</code>, <code>metrics</code>, <code>findings</code>, and the comparison block <code>comparisonSchemaVersion</code>, <code>stagePairs</code>, <code>unmatched</code>, <code>replanned</code>, <code>bookkeepingStageIds</code>, <code>executionAlignment</code> (built by the aligner, see <a href="./../state-and-history/run-comparison-internals.html">Run comparison internals</a>). <code>comparisonOutput()</code> takes a <code>view</code>: <code>full</code> (the CLI) carries every field, <code>summary</code> (MCP&#39;s default) leaves out <code>stagePairs</code>, which <code>compare_runs</code> adds back for <code>include: [&#39;stagePairs&#39;]</code>. <code>buildComparisonOutput()</code> also takes <code>normalizePath</code>, the CLI&#39;s repeatable <code>--normalize-path</code> and <code>compare_runs</code>&#39; <code>normalizePath</code>: it reaches <code>compareRuns</code> the way <code>thresholds</code> reaches <code>analyze()</code> and stays out of the run cache key, because it only affects the comparison. The CLI validates the patterns before any log is parsed (exit 2) and MCP before any run is resolved. That projection is the CLI&#39;s <code>comparison</code> object and the shared part of <code>compare_runs</code>. <code>compare_runs</code> also returns <code>metricDeltas</code> and <code>findingsDelta</code>, deprecated aliases of <code>metrics</code> and <code>findings</code> kept for one release.</li><li><code>runOutputBlocks()</code> in <code>packages/core/src/run-output.ts</code> builds the report&#39;s <code>metrics</code> and <code>effectiveConf</code> blocks, redacted from the redacted run, for both the CLI report and <code>diagnose_run</code>, which also returns the report&#39;s <code>writeTargets</code>.</li></ul><p>Neither surface wraps the other: MCP handlers hold a run cache and report errors by code, while the CLI maps failures to exit codes and owns stdout. Adding a field to the projection adds it to both. <code>packages/cli/test/parity.test.js</code> runs the CLI&#39;s <code>main()</code> and the MCP tools through an in-memory client on public corpus logs, flattens both outputs and diffs their key sets and values. The differences that stay (the MCP run handles, the deprecated aliases, the CLI&#39;s <code>schemaVersion</code>) are in a commented allowlist in that file, and an entry that explains no difference fails the test.</p><p>Every tool failure comes back as <code>{isError: true, content: [...], structuredContent: {code}}</code>, never an HTTP-status-shaped error; <code>code</code> is one of the 5 existing SHS codes (<code>SHS_ERROR_CODES</code>, <code>packages/core/src/shs-request.js</code>) plus <code>run-not-found</code>, <code>finding-not-found</code>, <code>invalid-event-log</code> (also covers a nonexistent <code>path</code> source), <code>archive-too-large</code> (SHS archive over the <code>SPARKFORENSICS_MAX_ARCHIVE_BYTES</code> byte cap, default 1 GiB, because the MCP path buffers the whole archive in memory, unlike the streaming <code>/shs-proxy</code> route), <code>invalid-type</code>, <code>invalid-anchor</code> (documentation tools), <code>directory-not-found</code>, <code>invalid-date-filter</code>, and <code>invalid-shs-base-url</code> (<code>list_runs</code>). An error without a code reports <code>access-or-upstream-failure</code>. A stalled SHS archive body fails as <code>upstream-unreachable</code> after <code>SPARKFORENSICS_SHS_TIMEOUT_MS</code> (default 30 s) without data.</p><p><code>scripts/vendor-core.mjs</code> (shared by <code>packages/cli</code>, <code>packages/mcp</code>, and <code>packages/server</code>&#39;s <code>prepack</code> scripts) vendors <code>packages/core/src/</code> wholesale into each package&#39;s own <code>vendor-core/</code> at pack time, pre-stripping TypeScript to plain <code>.js</code> (Node&#39;s native TS stripping refuses to run on <code>.ts</code> files under <code>node_modules</code>, which is exactly where a published <code>vendor-core/</code> lands). Each package&#39;s bin/entry point resolves its needed module through <code>packages/core/src/load-vendored.js</code>: from <code>vendor-core/</code> if present, else from the real <code>packages/core/src/</code> sibling loaded as <code>.ts</code> directly. In a monorepo checkout a leftover <code>vendor-core/</code> is used only while its <code>core-source-hash.txt</code> (written by <code>vendor-core.mjs</code>) matches <code>packages/core/src</code>; otherwise the bin warns on stderr and loads <code>packages/core/src</code> (<code>packages/server/index.js</code> mirrors <code>resolveStaticRoot</code>&#39;s <code>public/</code>-vs-<code>../dist</code> pattern for this same fallback).</p>',15)])])}const g=o(t,[["render",s]]);export{p as __pageData,g as default};
+import{_ as o,o as c,c as d,a5 as a}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"CLI and MCP server","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/worker-protocol/cli-and-mcp.md","filePath":"contributor-guide/architecture/worker-protocol/cli-and-mcp.md"}'),t={name:"contributor-guide/architecture/worker-protocol/cli-and-mcp.md"};function s(r,e,n,i,l,h){return c(),d("div",null,[...e[0]||(e[0]=[a('<h1 id="cli-and-mcp-server" tabindex="-1">CLI and MCP server <a class="header-anchor" href="#cli-and-mcp-server" aria-label="Permalink to &quot;CLI and MCP server&quot;">​</a></h1><p>The headless analysis CLI and the MCP server, which run the same parser and detectors outside the browser.</p><h2 id="headless-analysis-cli" tabindex="-1">Headless analysis CLI <a class="header-anchor" href="#headless-analysis-cli" aria-label="Permalink to &quot;Headless analysis CLI&quot;">​</a></h2><p><code>packages/cli/bin/sparkforensics-analyze.mjs</code> runs the same parser + detector contracts outside the browser, for CI. It accepts a single event-log file or a rolling-log directory, drives <code>runParse</code>/<code>runParseFiles</code> (<code>packages/core/src/parser-worker.ts</code>) with a Node-only File-like shim (<code>nodeFileFromPath</code> in <code>packages/core/src/cli/collect-run.ts</code>), and writes the exact <code>buildEvidenceReport</code> JSON schema described above: one format, not a second. The shim reads each requested slice with a positioned <code>readSync</code>, so a local log is never loaded whole and has no 2 GiB size limit; <code>collectRun</code> closes every descriptor it opened once parsing settles, on success and on error. Alternatively, <code>--shs-base-url &lt;url&gt; --app-id &lt;id&gt; [--attempt-id &lt;id&gt;]</code> fetches the run from a Spark History Server instead (mutually exclusive with the positional file/directory argument), calling <code>resolveFromShs</code> (<code>packages/core/src/shs-load.ts</code>, shared with the MCP server&#39;s SHS source path below) directly; no dependency on the <code>packages/server</code> package. The three flags are checked first with <code>validateShsRequest</code> (<code>packages/core/src/shs-request.js</code>), and a malformed one is a usage error (exit <code>2</code>) before any fetch. A failed SHS fetch reports its message to <code>stderr</code> and exits <code>4</code>, same as a candidate file that can&#39;t be parsed.</p><p>Regression budgets beyond the <code>--max-regression-pct</code> pair come from repeated <code>--regression-budget &lt;metric&gt;:&lt;pct&gt;</code> flags and a <code>--budgets</code> file, parsed in <code>packages/core/src/cli/regression-budgets.ts</code> and passed to <code>evaluateBudgets()</code> as <code>regressionBudgets</code>; the legacy pair is one more budget, and a metric budgeted twice is a usage error. Per-stage budgets (<code>--stage-regression-budget</code>, <code>--stage-quality</code>; <code>stageRegressionBudgets</code> and <code>stageQualities</code> on MCP) read the aligner&#39;s <code>stagePairs</code> through <code>checkStageRegression</code> in <code>cli/budgets.ts</code> and never match stages themselves; <code>parseStageRegressionBudgetFlag</code> refuses the neutral <code>inputBytes</code> and <code>outputBytes</code>. With two or more positional candidates (or <code>--format ndjson</code>), the CLI&#39;s <code>runMultiLog</code> parses and analyzes the baseline once, then evaluates each candidate in turn and writes one NDJSON line per candidate (<code>log</code>, <code>status</code>, <code>exitCode</code>, <code>error</code>, <code>budgets</code>, <code>candidate</code>, <code>comparison</code>). A candidate that can&#39;t be read or parsed gets a <code>status: &quot;error&quot;</code> line with <code>exitCode</code> <code>4</code> (<code>6</code> for an internal failure on that candidate); the exit code is the worst line (<code>6</code> &gt; <code>5</code> &gt; <code>4</code> &gt; <code>1</code> &gt; <code>3</code> &gt; <code>0</code>, <code>EXIT_SEVERITY</code>). A baseline that can&#39;t be read exits <code>5</code> with no lines. Under <code>--redact</code>, <code>log</code> and the <code>stderr</code> prefixes are <code>candidate-&lt;n&gt;</code> (1-based argument position) and an error line&#39;s message is generic, so no candidate path, which usually carries the app id, reaches the output. In both modes <code>--redact</code> also swaps the <code>stderr</code> text of a baseline or candidate failure (SHS fetch included) for <code>redactedFailure</code>&#39;s role-only message.</p><p>Optional CLI-flag budgets (<code>--max-runtime &lt;ms&gt;</code>, <code>--max-spill &lt;gb&gt;</code>, <code>--max-skew &lt;ratio&gt;</code>, <code>--max-failed-task-rate &lt;pct&gt;</code>, <code>--min-efficiency &lt;pct&gt;</code>) are evaluated in <code>packages/core/src/cli/budgets.ts</code> against the existing finding catalog (<code>analyze()</code>) and <code>computeEfficiencyModel</code>; there is no second rule engine. <code>--min-efficiency</code> compares <code>100 - wastagePct</code> (busy core time, the complement of the dashboard&#39;s Unused core time) and says so in its detail (&quot;Busy core time 26% below budget 90%.&quot;); it is not the Scorecard&#39;s Efficiency tile (<code>stagesActive / total</code>). A budget whose required evidence is missing (e.g. the run never emitted <code>ApplicationEnd</code>, or has no usable per-task <code>runAggregates</code>) is reported as inconclusive (<code>stderr</code> warning) rather than silently passing, and gets its own exit code distinct from both pass and violation. Exit codes: <code>0</code> pass, <code>1</code> a configured budget was violated, <code>2</code> a usage error (unknown or value-less flag, unknown <code>--regression-metric</code> key, an unreadable or invalid <code>--thresholds</code> or <code>--budgets</code> file), <code>3</code> no violations but at least one budget was inconclusive, <code>4</code> the candidate could not be read or parsed (or its SHS fetch failed), <code>5</code> the <code>--baseline</code> could not be read or parsed, <code>6</code> an internal error (an unhandled failure, a failed <code>--export-html</code> export). <code>main</code> catches anything unhandled and exits <code>6</code> rather than letting Node exit <code>1</code>, which would read as a violation. When several apply the worst wins: <code>6</code>, <code>5</code>, <code>4</code>, <code>1</code>, <code>3</code>, <code>0</code>; a violation always wins over an inconclusive result in the same run (exit <code>1</code>, not <code>3</code>). <code>evaluateBudgets()</code> also always adds an inconclusive <code>run-complete</code> result when the catalog has an <code>incompleteRun</code> finding, so a run with no <code>ApplicationEnd</code> exits <code>3</code> even with no budget flags. With <code>--baseline</code>, the absolute budgets and this check apply to the candidate run; the MCP <code>evaluate_budgets</code> tool below uses the same function with the same roles.</p><h2 id="mcp-server" tabindex="-1">MCP server <a class="header-anchor" href="#mcp-server" aria-label="Permalink to &quot;MCP server&quot;">​</a></h2><p><code>packages/core/src/mcp-server-factory.ts</code>&#39;s <code>createMcpServer()</code> registers 8 tools: <code>list_runs</code> (candidate runs in a local directory or on a Spark History Server, to pick one before diagnosing it), <code>diagnose_run</code> (thresholded findings + remediation text), <code>get_run_summary</code> (app/stage/job/sql counts and duration, no findings), <code>compare_runs</code> (the comparison verdict from <code>comparisonVerdict</code> in <code>packages/core/src/comparison-verdict.ts</code>, the dashboard comparison page&#39;s own headline, plus categorized findings delta + metric deltas between two runs; <code>CompareRunsResult.jobOutcomes</code> carries each run&#39;s failed jobs and incomplete flag for it), <code>evaluate_budgets</code> (pass/fail budget thresholds against one run, optionally with a second run for regression/fail-on-introduced budgets: the MCP side of the CLI&#39;s <code>evaluateBudgets()</code> gating), <code>get_finding_evidence</code> (raw evidence bundle for one finding, for drill-down after <code>diagnose_run</code>), <code>get_finding_documentation</code> (detection/tuning reference docs for one finding type, independent of any run), and <code>get_reference_doc</code> (a full tuning-reference chapter or bottleneck page by doc anchor, e.g. <code>#joins</code>). None re-implement detector logic: <code>list_runs</code> lists candidates (<code>packages/core/src/list-runs.ts</code>), and the rest repackage <code>analyze</code>/<code>buildEvidenceReport</code>/<code>compareRuns</code>/<code>captureSnapshot</code>/<code>evaluateBudgets</code> from <code>packages/core/src/mcp-tools.ts</code>, which resolves a <code>source</code> (event-log <code>path</code>, or an SHS <code>shsBaseUrl</code>/<code>appId</code>/<code>attemptId</code> triple) into a cached <code>AppModel</code>. The SHS source path calls <code>resolveFromShs</code> (<code>packages/core/src/shs-load.ts</code>, also used directly by the CLI&#39;s <code>--shs-base-url</code> mode above), which reuses two extractions shared with the browser ingestion flow: <code>fetchShsEventLog</code> (<code>packages/core/src/proxy.js</code>, the same upstream fetch the local server&#39;s <code>/shs-proxy</code> route uses) fetches the zip, and <code>decodeShsArchive</code> (<code>packages/core/src/shs-fetch.ts</code>, re-exported from <code>parser-worker.ts</code>&#39;s barrel) streams it through the parser as worker messages, which <code>collectViaDispatch</code> assembles into an <code>AppModel</code>. <code>runParseFromUrl</code> calls the same <code>decodeShsArchive</code> after fetching through the proxy.</p><p>Two transports connect to that one factory: <code>packages/mcp/bin/sparkforensics-mcp.mjs</code> (stdio, for local MCP clients) and <code>packages/server/index.js</code>&#39;s <code>/mcp</code> route (streamable HTTP, for the local server). The run cache (<code>packages/core/src/mcp-tools.ts</code>) is a module-level LRU (cap 8 and 15-minute idle TTL by default, overridable via <code>SPARKFORENSICS_MCP_CACHE_CAP</code> and <code>SPARKFORENSICS_MCP_CACHE_TTL_MS</code>, lazily swept on access) keyed by resolved source (path plus mtime, ctime and size, or SHS baseUrl+appId+attemptId), so a client mints a <code>runId</code> once via <code>resolveOrCreateRun</code> and reuses it across subsequent tool calls instead of re-parsing.</p><h3 id="one-output-builder-for-the-cli-and-mcp" tabindex="-1">One output builder for the CLI and MCP <a class="header-anchor" href="#one-output-builder-for-the-cli-and-mcp" aria-label="Permalink to &quot;One output builder for the CLI and MCP&quot;">​</a></h3><p>Neither surface lists the fields of a run or a comparison by hand. The CLI&#39;s JSON and the MCP tools call the same core functions:</p><ul><li><code>buildComparisonOutput()</code> in <code>packages/core/src/comparison-output.ts</code> diffs the two runs, applies <code>--redact</code>/<code>redact</code> itself (so no surface can budget or report an unredacted comparison by skipping it), and returns the redacted <code>CompareRunsResult</code> for the budget checks and the Markdown renderer, plus <code>comparisonOutput()</code>&#39;s projection: <code>verdict</code>, <code>confidence</code>, <code>reason</code>, <code>matchedCoverage</code>, <code>runtimeCoverage</code>, <code>metrics</code>, <code>findings</code>, and the comparison block <code>comparisonSchemaVersion</code>, <code>stagePairs</code>, <code>unmatched</code>, <code>replanned</code>, <code>bookkeepingStageIds</code>, <code>executionAlignment</code> (built by the aligner, see <a href="./../state-and-history/run-comparison-internals.html">Run comparison internals</a>). <code>comparisonOutput()</code> takes a <code>view</code>: <code>full</code> (the CLI) carries every field, <code>summary</code> (MCP&#39;s default) leaves out <code>stagePairs</code>, which <code>compare_runs</code> adds back for <code>include: [&#39;stagePairs&#39;]</code>. <code>buildComparisonOutput()</code> also takes <code>normalizePath</code>, the CLI&#39;s repeatable <code>--normalize-path</code> and <code>compare_runs</code>&#39; <code>normalizePath</code>: it reaches <code>compareRuns</code> the way <code>thresholds</code> reaches <code>analyze()</code> and stays out of the run cache key, because it only affects the comparison. The CLI validates the patterns before any log is parsed (exit 2) and MCP before any run is resolved. That projection is the CLI&#39;s <code>comparison</code> object and the shared part of <code>compare_runs</code>. <code>compare_runs</code> also returns <code>metricDeltas</code> and <code>findingsDelta</code>, deprecated aliases of <code>metrics</code> and <code>findings</code> kept for one release.</li><li><code>runOutputBlocks()</code> in <code>packages/core/src/run-output.ts</code> builds the report&#39;s <code>metrics</code> and <code>effectiveConf</code> blocks, redacted from the redacted run, for both the CLI report and <code>diagnose_run</code>, which also returns the report&#39;s <code>writeTargets</code>. It also returns the <code>generator</code> block when the surface passes one. Core does not know its host, so the CLI supplies its package name, version and core build id (<code>coreBuildId</code>), and MCP supplies none: the server factory has no real package version to report.</li></ul><p>Neither surface wraps the other: MCP handlers hold a run cache and report errors by code, while the CLI maps failures to exit codes and owns stdout. Adding a field to the projection adds it to both. <code>packages/cli/test/parity.test.js</code> runs the CLI&#39;s <code>main()</code> and the MCP tools through an in-memory client on public corpus logs, flattens both outputs and diffs their key sets and values. The differences that stay (the MCP run handles, the deprecated aliases, the CLI&#39;s <code>schemaVersion</code>) are in a commented allowlist in that file, and an entry that explains no difference fails the test.</p><p>Every tool failure comes back as <code>{isError: true, content: [...], structuredContent: {code}}</code>, never an HTTP-status-shaped error; <code>code</code> is one of the 5 existing SHS codes (<code>SHS_ERROR_CODES</code>, <code>packages/core/src/shs-request.js</code>) plus <code>run-not-found</code>, <code>finding-not-found</code>, <code>invalid-event-log</code> (also covers a nonexistent <code>path</code> source), <code>archive-too-large</code> (SHS archive over the <code>SPARKFORENSICS_MAX_ARCHIVE_BYTES</code> byte cap, default 1 GiB, because the MCP path buffers the whole archive in memory, unlike the streaming <code>/shs-proxy</code> route), <code>invalid-type</code>, <code>invalid-anchor</code> (documentation tools), <code>directory-not-found</code>, <code>invalid-date-filter</code>, and <code>invalid-shs-base-url</code> (<code>list_runs</code>). An error without a code reports <code>access-or-upstream-failure</code>. A stalled SHS archive body fails as <code>upstream-unreachable</code> after <code>SPARKFORENSICS_SHS_TIMEOUT_MS</code> (default 30 s) without data.</p><p><code>scripts/vendor-core.mjs</code> (shared by <code>packages/cli</code>, <code>packages/mcp</code>, and <code>packages/server</code>&#39;s <code>prepack</code> scripts) vendors <code>packages/core/src/</code> wholesale into each package&#39;s own <code>vendor-core/</code> at pack time, pre-stripping TypeScript to plain <code>.js</code> (Node&#39;s native TS stripping refuses to run on <code>.ts</code> files under <code>node_modules</code>, which is exactly where a published <code>vendor-core/</code> lands). Each package&#39;s bin/entry point resolves its needed module through <code>packages/core/src/load-vendored.js</code>: from <code>vendor-core/</code> if present, else from the real <code>packages/core/src/</code> sibling loaded as <code>.ts</code> directly. In a monorepo checkout a leftover <code>vendor-core/</code> is used only while its <code>core-source-hash.txt</code> (written by <code>vendor-core.mjs</code>) matches <code>packages/core/src</code>; otherwise the bin warns on stderr and loads <code>packages/core/src</code> (<code>packages/server/index.js</code> mirrors <code>resolveStaticRoot</code>&#39;s <code>public/</code>-vs-<code>../dist</code> pattern for this same fallback).</p>',15)])])}const g=o(t,[["render",s]]);export{p as __pageData,g as default};

package/public/docs/assets/{contributor-guide_architecture_worker-protocol_evidence-report.md.DVGZOWvW.js → contributor-guide_architecture_worker-protocol_evidence-report.md.CiQl1MLb.js}

@@ -1 +1 @@
-import{_ as o,o as d,c as t,a5 as c}from"./chunks/framework.DSg0KOwT.js";const u=JSON.parse('{"title":"Evidence report and finding identity","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/worker-protocol/evidence-report.md","filePath":"contributor-guide/architecture/worker-protocol/evidence-report.md"}'),a={name:"contributor-guide/architecture/worker-protocol/evidence-report.md"};function n(i,e,r,s,l,h){return d(),t("div",null,[...e[0]||(e[0]=[c('<h1 id="evidence-report-and-finding-identity" tabindex="-1">Evidence report and finding identity <a class="header-anchor" href="#evidence-report-and-finding-identity" aria-label="Permalink to &quot;Evidence report and finding identity&quot;">​</a></h1><p>The portable evidence report and the stable identity every finding carries.</p><h2 id="portable-evidence-report" tabindex="-1">Portable evidence report <a class="header-anchor" href="#portable-evidence-report" aria-label="Permalink to &quot;Portable evidence report&quot;">​</a></h2><p><code>packages/core/src/evidence-report.ts</code>&#39;s <code>buildEvidenceReport(appModel, { redact })</code> returns <code>{ markdown, json }</code>: a self-contained, byte-stable document that runs the detectors over an <code>appModel</code> and serializes the result for sharing outside the tool. Raw task records are never included; identifier redaction (app id + host names → <code>app-1</code>/<code>host-1</code> pseudonyms via <code>packages/core/src/redact.ts</code>, with the app name and <code>spark.app.name</code> replaced by the app id&#39;s pseudonym) is opt-in with <code>{ redact: true }</code>. The JSON is pinned by <code>EVIDENCE_SCHEMA_VERSION</code> (<code>packages/core/src/evidence-report.ts</code>, surfaced as <code>json.schemaVersion</code>) and has this fixed top-level key order:</p><div class="language-text vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">text</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span>schemaVersion, summary, verdict, evidenceAvailability, detectors, findings, writeTargets, recommendations, cleanChecks, notRunChecks</span></span></code></pre></div><ul><li><code>summary</code> is the run header: <code>{ app: { id, name, sparkVersion }, stageCount, jobCount, sqlExecutionCount, findingCount, impactBandCounts, actionableFindingCount, actionableImpactBandCounts, clean, outcome, runShape }</code>. <code>findingCount</code> counts every row in <code>findings</code>; the <code>actionable*</code> counts leave out evidence caveats and the <code>incompleteRun</code> row, the same set the dashboard&#39;s top bar and verdict count. <code>clean</code> is <code>isCleanRun</code> from <code>packages/core/src/check-coverage.ts</code>, and <code>outcome</code> is <code>{ failedJobs, totalJobs, failureReason, failureReasonStageId }</code> from <code>summarizeRunOutcome</code> (<code>packages/core/src/run-outcome.ts</code>), the job results the dashboard verdict leads with; the Markdown prints it as an <code>- Outcome:</code> header line. <code>runShape</code> (<code>computeRunShape</code>, <code>packages/core/src/run-shape.ts</code>) is the Scorecard&#39;s wall-clock, Efficiency and Unused core time, the ETL phases&#39; summed stage time and Core Usage by Locality&#39;s peak busy cores, each <code>null</code> where the dashboard shows none; MCP <code>get_run_summary</code> returns it too.</li><li><code>verdict</code> is the dashboard&#39;s run verdict from <code>buildRunVerdict</code> (<code>packages/core/src/run-verdict.ts</code>): <code>{ title, summary, steps, remainingPlaces, copyText }</code>. <code>steps</code> holds the first three places to look in the verdict card&#39;s order, each <code>{ key, stageId, type, tag, leadFindingId, actionLabel, recommendation, impact, impactMeaning, relatedTypes, text }</code>, where <code>text</code> is that step&#39;s line of the &quot;Copy next steps&quot; checklist and <code>copyText</code> is the whole checklist. The Markdown opens with it as <code>## Verdict</code>. The <code>recommendations</code> rollup stays alongside: it ranks fix types, the verdict ranks places.</li><li><code>evidenceAvailability</code> is the ledger above (or <code>null</code> when absent).</li><li><code>detectors</code> is <code>detectorCatalog()</code> output: one <code>{ type, version, scope, thresholds, docAnchor }</code> per detector, in <code>DETECTORS</code> order, so the exact threshold set that produced each finding travels with the evidence. On a tuned run (see the tuned-run paragraph below) a tuned row&#39;s <code>thresholds</code> are the ones the run used, plus <code>tunedThresholds</code>.</li><li><code>findings</code> are deterministically sorted rows (impact band → type → stage → id), each with a stable <code>id</code>, <code>tag</code>, core columns, an always-present <code>actionLabel</code>, and an <code>evidence</code> sub-object holding that finding type&#39;s declared evidence fields (see the per-type evidence paragraph below); <code>confidence</code>/ <code>validationRequired</code>/<code>docAnchor</code> appear only when the detector emitted them.</li><li><code>writeTargets</code> is <code>extractWriteTargets</code> output (<code>packages/core/src/write-targets.ts</code>): every SQL write command with its path or table. The field contract is in the user guide&#39;s <a href="./../../../user-guide/getting-started/ci-and-automation.html#write-targets">Write targets</a>.</li><li><code>recommendations</code> is the impact-ranked <code>buildRecommendationRollup</code> output (<code>packages/core/src/recommendation-rollup.ts</code>), and <code>cleanChecks</code> lists every detector type that fired zero findings this run and could run: see the rollup and clean-checks paragraph below.</li><li><code>notRunChecks</code> lists the zero-finding types the log lacked the data to run, each with a <code>reason</code>: see the not-run checks paragraph below.</li></ul><p>Determinism holds because detector order, finding sort, and object key order are all fixed, so a given <code>appModel</code> serializes identically across calls.</p><p>Redaction enumerates hosts two ways: by walking the findings tree for every string value under a key literally named <code>host</code> (<code>evidence.host</code>, <code>evidence.failedTaskDetails[].host</code>, <code>evidence.retriedTaskDetails[].host</code>, and any nested <code>host</code> field, all covered without enumerating paths), and by scanning every string value for EC2-style hostnames / bare IPv4 tokens. So identifiers that surface only in free text (recommendation copy, a <code>stageFailed</code> failure reason in <code>valueText</code>) are pseudonymized too. Pseudonym numbering uses a numeric-aware sort, so re-redacting an already-redacted report is a no-op even past <code>host-10</code>.</p><p>Failed-task error text is dropped rather than pseudonymized, since a message or stack trace can carry file paths and data values that no host pattern matches: every array under a key named <code>failureGroups</code> has its <code>message</code> replaced and the message text stripped from its <code>stackExcerpt</code> (<code>redactTaskFailureGroup</code> in <code>task-failure.ts</code>). That covers the evidence report and both the findings and the stage records of the HTML export. A stage&#39;s failure reason is replaced outright with <code>[redacted]</code> wherever it appears: a <code>stageFailed</code> finding&#39;s <code>valueText</code>, a stage record&#39;s <code>stageFailureReason</code>, a job&#39;s <code>exception</code> and the report&#39;s <code>failureReason</code> (<code>redactStageFailureReasons</code> in <code>redact.ts</code>).</p><p>The Markdown rendering mirrors the JSON&#39;s field set: each finding block prints its <code>detector version</code>, its sorted <code>evidence</code> entries (byte-magnitude keys humanized), and the report ends with a <code>## Detectors</code> catalog carrying the version + threshold set. A finding&#39;s <code>impactEstimate</code> prints as its own <code>- impact: </code> line worded as the dashboard&#39;s &quot;Potential savings&quot;: the wall-clock range, or the raw waste only when there is no range, followed by what it counts (&quot;of run time&quot;, &quot;of unused executor memory&quot;) and <code>estimateMethod</code>, and nothing for an informational or zero estimate. A <code>- estimate: </code> line carries the Advanced view&#39;s provenance sentence (<code>estimateProvenance</code>). Both come from <code>packages/core/src/impact-format.ts</code>, the formatters every dashboard surface uses, so memory reads in GB-h from 0.1 GB-h up, core time in core-s or core-h, and a time figure has no &quot;Estimated&quot; prefix. JSON finding rows carry the same figure as <code>impact</code>/<code>impactMeaning</code> when there is one, and each <code>recommendations</code> row an <code>impactMeaning</code> next to its <code>impact</code>, which is null for a resource figure that rounds to zero. <code>EvidenceExport</code> names downloads <code>evidence-&lt;appId&gt;[-redacted].&lt;md|json|html&gt;</code>, taking the app id from the (already pseudonymized when redacting) report so a redacted file never leaks the real id and is never name-identical to a raw export.</p><p>A finding row carries <code>impactEstimate</code> when the finding has one, with the full contract documented in <a href="./../impact-estimation.html#occupancy-weighted-attribution">Impact estimation</a> (basis, wallClock, estimateMethod, rawWaste, coreTimeMs). It appears after the pinned core columns (<code>id</code>, <code>type</code>, <code>impactBand</code>, <code>stageId</code>, <code>metric</code>, <code>value</code>, <code>recommendation</code>, <code>detectorVersion</code>, plus optional but pinned <code>confidence</code>, <code>validationRequired</code>, <code>docAnchor</code>) without displacing any of them. The full row shape is <code>FindingRowColumns</code> in <code>evidence-report.ts</code>, which also carries <code>name</code>, <code>tag</code>, <code>valueText</code>, <code>remediation</code>, <code>actionLabel</code>, <code>impact</code>/<code>impactMeaning</code> and, on a tuned run, <code>tunedThresholds</code>.</p><p>The report carries the same recommendation data the dashboard&#39;s &quot;Fix These First&quot; view shows, so the CLI (<code>packages/cli/bin/sparkforensics-analyze.mjs</code>) and the MCP tool (<code>diagnoseRun</code> in <code>packages/core/src/mcp-tools.ts</code>) get it too, not just the web markdown/JSON download. <code>FindingRow.actionLabel</code> is always present (a short imperative label like &quot;Reduce shuffle size&quot;), from <code>findingActionLabel</code> (<code>packages/core/src/finding-action-label.ts</code>), which reads the type&#39;s <code>actionLabel</code> in <code>finding-presentation.ts</code> and falls back to the type&#39;s name. The dashboard, the run verdict and the report rows all call it. <code>EvidenceReportJson.recommendations</code> is the same impact-ranked <code>buildRecommendationRollup</code> grouping (<code>packages/core/src/recommendation-rollup.ts</code>) that <code>FixTheseFirst.tsx</code> renders, so CLI/MCP/download consumers get the same &quot;what&#39;s the highest-impact fix&quot; ranking the dashboard shows, without changing the <code>findings</code> array&#39;s own impact-band-sorted order at all. <code>EvidenceReportJson.cleanChecks</code> lists every detector type that fired zero findings this run, each with <code>getThresholdSummary</code>&#39;s one-line &quot;what would have tripped it&quot; sentence; unlike the dashboard&#39;s <code>Alerts.tsx</code> clean-checks table, it deliberately includes the one &quot;always-mounted&quot; reference type (<code>coreLocality</code>) even when it has no findings, since a flat evidence report has no separate always-visible surface for it to already appear on the way that widget does on the board.</p><p>A check the log could not run is not listed as clean. <code>packages/core/src/check-coverage.ts</code> holds the one rule, shared with the dashboard&#39;s verdict, top bar and Clean checks: a type whose only finding is an evidence caveat, every <code>scope: &#39;stage&#39;</code> type on a log where no stage recorded an end, and the run-span types (<code>utilization</code>, <code>memoryUtilization</code>, <code>autoscalingChurn</code>) on a log with no ApplicationEnd. Those types are listed in <code>notRunChecks</code> instead of <code>cleanChecks</code>, each <code>{ type, tag, thresholdSummary, reason }</code>, where <code>reason</code> is the caveat&#39;s own recommendation (it names the setting to turn on) or the log-wide sentence. The Markdown has a <code>## Not checked on this log</code> section above <code>## Clean checks</code>, and a <code>Findings to act on</code> header line.</p><p><code>Finding</code> is a union discriminated on <code>type</code>, one member per emitted finding type (<code>packages/core/src/finding-types.ts</code>), and a finding row&#39;s <code>evidence</code> is an explicit per-type projection. Each type&#39;s <code>&lt;Type&gt;Evidence</code> interface names its public fields, and <code>EVIDENCE_KEYS</code> in <code>evidence-report.ts</code> lists the same keys, checked both ways at compile time, so a field a detector adds only for another core module does not become report contract. Rows leave out these fields: <code>stageShape</code>&#39;s <code>totalCores</code>; <code>utilization</code>&#39;s <code>utilizationFraction</code>, <code>appDurationMs</code> and <code>totalCores</code>; <code>memoryUtilization</code>&#39;s <code>idleRateFraction</code>, <code>allocatedMB</code>, <code>peakExecutors</code>, <code>appDurationMs</code> and <code>allocatedBytes</code>; <code>retryWaste</code>&#39;s <code>extended</code> display copy. The impact estimator reads them on the finding. <code>value</code> is always numeric or <code>null</code>: the text-valued findings (<code>stageFailed</code>&#39;s failure reason, <code>configAudit</code>&#39;s current setting, <code>incompleteRun</code>&#39;s <code>missing</code>) carry their text in a <code>valueText</code> column, present only on those rows, and the Markdown prints it where <code>value</code> would go. Renaming or removing an evidence field is a breaking change and needs a schema bump. <code>cleanChecks</code>/<code>notRunChecks</code> list emitted finding types, the set the dashboard&#39;s Clean checks shows: <code>overBroadcast</code> and <code>underBroadcast</code> in place of the <code>broadcastSizing</code> detector entry. A finding row&#39;s <code>actionLabel</code> for a (type, discriminant) combination with no label of its own is the type&#39;s name, the same fallback the verdict step uses.</p><p><code>buildEvidenceReport(appModel, { thresholds })</code> runs the detectors with a user&#39;s validated overrides (the CLI&#39;s and MCP server&#39;s <code>--thresholds</code>; see <a href="./../detector-contract.html#tuning-thresholds">Tuning thresholds</a>). Only a tuned run adds keys: <code>tunedThresholds</code> (<code>{ &lt;name&gt;: { value, default } }</code>) on each finding row and each <code>cleanChecks</code>/<code>notRunChecks</code> entry from a detector an override moved off its defaults (or whose <code>suppressedBy</code> detector it moved), on that detector&#39;s <code>detectors</code> row, and as <code>summary.tunedThresholds</code> keyed by detector type. The Markdown adds a <code>- Tuned thresholds:</code> header line, a <code>- tuned thresholds:</code> line per affected finding, and marks tuned catalog rows and clean checks. A default run&#39;s report carries none of these keys, and every one is optional. The report caches key on the overrides object as well as the <code>appModel</code>, so one model&#39;s default and tuned reports never mix. Finding ids ignore thresholds: a finding a tuned run still emits keeps the id it has in a default run.</p><h2 id="finding-identity" tabindex="-1">Finding identity <a class="header-anchor" href="#finding-identity" aria-label="Permalink to &quot;Finding identity&quot;">​</a></h2><p>Every finding <code>analyzer.ts</code> emits carries a stable <code>id</code> (<code>push()</code>&#39;s <code>findingId()</code> choke point, FNV-1a hash over <code>type | location | metric | value | discriminators</code>) and a <code>detectorVersion</code> (from the emitting <code>DETECTORS</code> entry). The two are deliberately decoupled: <code>id</code> is derived only from a finding&#39;s evidence tuple, never from <code>detectorVersion</code>, so a threshold or logic tweak that bumps a detector&#39;s <code>version</code> does not change the <code>id</code> of findings it still emits at the same location/metric. A saved reference (dashboard bookmark, exported report row) keeps pointing at the same logical finding across detector revisions; <code>detectorVersion</code> is separate provenance metadata for &quot;which ruleset produced this,&quot; not part of identity. See <code>packages/core/test/analyzer-finding-identity.test.js</code> for the decoupling proof.</p><p>The discriminators are declared per finding type (<code>ID_DISCRIMINATORS</code> in <code>analyzer.ts</code>, each key checked against that type&#39;s fields) and joined in the fixed <code>DISCRIMINATOR_SLOTS</code> order, and the value slot takes <code>value</code>, else <code>valueText</code>.</p><p><code>id</code> stability holds for equivalent reruns of the same schema/analyzer version on the same input. Changing the id-derivation rule itself (the hash algorithm, or which fields feed the location key/discriminators) is an intentional breaking change and must bump <code>EVIDENCE_SCHEMA_VERSION</code> (<code>packages/core/src/evidence-report.ts</code>); there is no separate id-scheme version.</p>',19)])])}const m=o(a,[["render",n]]);export{u as __pageData,m as default};
+import{_ as o,o as d,c as t,a5 as c}from"./chunks/framework.DSg0KOwT.js";const u=JSON.parse('{"title":"Evidence report and finding identity","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/worker-protocol/evidence-report.md","filePath":"contributor-guide/architecture/worker-protocol/evidence-report.md"}'),a={name:"contributor-guide/architecture/worker-protocol/evidence-report.md"};function n(i,e,r,s,l,h){return d(),t("div",null,[...e[0]||(e[0]=[c('<h1 id="evidence-report-and-finding-identity" tabindex="-1">Evidence report and finding identity <a class="header-anchor" href="#evidence-report-and-finding-identity" aria-label="Permalink to &quot;Evidence report and finding identity&quot;">​</a></h1><p>The portable evidence report and the stable identity every finding carries.</p><h2 id="portable-evidence-report" tabindex="-1">Portable evidence report <a class="header-anchor" href="#portable-evidence-report" aria-label="Permalink to &quot;Portable evidence report&quot;">​</a></h2><p><code>packages/core/src/evidence-report.ts</code>&#39;s <code>buildEvidenceReport(appModel, { redact })</code> returns <code>{ markdown, json }</code>: a self-contained, byte-stable document that runs the detectors over an <code>appModel</code> and serializes the result for sharing outside the tool. Raw task records are never included; identifier redaction (app id + host names → <code>app-1</code>/<code>host-1</code> pseudonyms via <code>packages/core/src/redact.ts</code>, with the app name and <code>spark.app.name</code> replaced by the app id&#39;s pseudonym) is opt-in with <code>{ redact: true }</code>. The JSON is pinned by <code>EVIDENCE_SCHEMA_VERSION</code> (<code>packages/core/src/evidence-report.ts</code>, surfaced as <code>json.schemaVersion</code>) and has this fixed top-level key order:</p><div class="language-text vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">text</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span>schemaVersion, summary, verdict, evidenceAvailability, detectors, findings, writeTargets, recommendations, cleanChecks, notRunChecks</span></span></code></pre></div><ul><li><code>summary</code> is the run header: <code>{ app: { id, name, sparkVersion }, stageCount, jobCount, sqlExecutionCount, findingCount, impactBandCounts, actionableFindingCount, actionableImpactBandCounts, clean, outcome, runShape }</code>. <code>findingCount</code> counts every row in <code>findings</code>; the <code>actionable*</code> counts leave out evidence caveats and the <code>incompleteRun</code> row, the same set the dashboard&#39;s top bar and verdict count. <code>clean</code> is <code>isCleanRun</code> from <code>packages/core/src/check-coverage.ts</code>, and <code>outcome</code> is <code>{ failedJobs, totalJobs, failureReason, failureReasonStageId }</code> from <code>summarizeRunOutcome</code> (<code>packages/core/src/run-outcome.ts</code>), the job results the dashboard verdict leads with; the Markdown prints it as an <code>- Outcome:</code> header line. <code>runShape</code> (<code>computeRunShape</code>, <code>packages/core/src/run-shape.ts</code>) is the Scorecard&#39;s wall-clock, Efficiency and Unused core time, the ETL phases&#39; summed stage time and Core Usage by Locality&#39;s peak busy cores, each <code>null</code> where the dashboard shows none; MCP <code>get_run_summary</code> returns it too.</li><li><code>verdict</code> is the dashboard&#39;s run verdict from <code>buildRunVerdict</code> (<code>packages/core/src/run-verdict.ts</code>): <code>{ title, summary, steps, remainingPlaces, copyText }</code>. <code>steps</code> holds the first three places to look in the verdict card&#39;s order, each <code>{ key, stageId, type, tag, leadFindingId, actionLabel, recommendation, impact, impactMeaning, relatedTypes, text }</code>, where <code>text</code> is that step&#39;s line of the &quot;Copy next steps&quot; checklist and <code>copyText</code> is the whole checklist. The Markdown opens with it as <code>## Verdict</code>. The <code>recommendations</code> rollup stays alongside: it ranks fix types, the verdict ranks places.</li><li><code>evidenceAvailability</code> is the ledger above (or <code>null</code> when absent).</li><li><code>detectors</code> is <code>detectorCatalog()</code> output: one <code>{ type, version, scope, thresholds, docAnchor }</code> per detector, in <code>DETECTORS</code> order, so the exact threshold set that produced each finding travels with the evidence. On a tuned run (see the tuned-run paragraph below) a tuned row&#39;s <code>thresholds</code> are the ones the run used, plus <code>tunedThresholds</code>.</li><li><code>findings</code> are deterministically sorted rows (impact band → type → stage → id), each with a stable <code>id</code>, <code>tag</code>, core columns, an always-present <code>actionLabel</code>, and an <code>evidence</code> sub-object holding that finding type&#39;s declared evidence fields (see the per-type evidence paragraph below); <code>confidence</code>/ <code>validationRequired</code>/<code>docAnchor</code> appear only when the detector emitted them.</li><li><code>writeTargets</code> is <code>extractWriteTargets</code> output (<code>packages/core/src/write-targets.ts</code>): every SQL write command with its path or table. The field contract is in the user guide&#39;s <a href="./../../../user-guide/getting-started/ci-and-automation.html#write-targets">Write targets</a>.</li><li><code>recommendations</code> is the impact-ranked <code>buildRecommendationRollup</code> output (<code>packages/core/src/recommendation-rollup.ts</code>), and <code>cleanChecks</code> lists every detector type that fired zero findings this run and could run: see the rollup and clean-checks paragraph below.</li><li><code>notRunChecks</code> lists the zero-finding types the log lacked the data to run, each with a <code>reason</code>: see the not-run checks paragraph below.</li></ul><p>Determinism holds because detector order, finding sort, and object key order are all fixed, so a given <code>appModel</code> serializes identically across calls.</p><p>Redaction enumerates hosts two ways: by walking the findings tree for every string value under a key literally named <code>host</code> (<code>evidence.host</code>, <code>evidence.failedTaskDetails[].host</code>, <code>evidence.retriedTaskDetails[].host</code>, and any nested <code>host</code> field, all covered without enumerating paths), and by scanning every string value for EC2-style hostnames / bare IPv4 tokens. So identifiers that surface only in free text (recommendation copy, a <code>stageFailed</code> failure reason in <code>valueText</code>) are pseudonymized too. Pseudonym numbering uses a numeric-aware sort, so re-redacting an already-redacted report is a no-op even past <code>host-10</code>.</p><p>Failed-task error text is dropped rather than pseudonymized, since a message or stack trace can carry file paths and data values that no host pattern matches: every array under a key named <code>failureGroups</code> has its <code>message</code> replaced and the message text stripped from its <code>stackExcerpt</code> (<code>redactTaskFailureGroup</code> in <code>task-failure.ts</code>). That covers the evidence report and both the findings and the stage records of the HTML export. A stage&#39;s failure reason is replaced outright with <code>[redacted]</code> wherever it appears: a <code>stageFailed</code> finding&#39;s <code>valueText</code>, a stage record&#39;s <code>stageFailureReason</code>, a job&#39;s <code>exception</code> and the report&#39;s <code>failureReason</code> (<code>redactStageFailureReasons</code> in <code>redact.ts</code>).</p><p>The Markdown rendering mirrors the JSON&#39;s field set: each finding block prints its <code>detector version</code>, its sorted <code>evidence</code> entries (byte-magnitude keys humanized), and the report ends with a <code>## Detectors</code> catalog carrying the version + threshold set. A finding&#39;s <code>impactEstimate</code> prints as its own <code>- impact: </code> line worded as the dashboard&#39;s &quot;Potential savings&quot;: the wall-clock range, or the raw waste only when there is no range, followed by what it counts (&quot;of run time&quot;, &quot;of unused executor memory&quot;) and <code>estimateMethod</code>, and nothing for an informational or zero estimate. A <code>- estimate: </code> line carries the Advanced view&#39;s provenance sentence (<code>estimateProvenance</code>). Both come from <code>packages/core/src/impact-format.ts</code>, the formatters every dashboard surface uses, so memory reads in GB-h from 0.1 GB-h up, core time in core-s or core-h, and a time figure has no &quot;Estimated&quot; prefix. JSON finding rows carry the same figure as <code>impact</code>/<code>impactMeaning</code> when there is one, and each <code>recommendations</code> row an <code>impactMeaning</code> next to its <code>impact</code>, which is null for a resource figure that rounds to zero. <code>EvidenceExport</code> names downloads <code>evidence-&lt;appId&gt;[-redacted].&lt;md|json|html&gt;</code>, taking the app id from the (already pseudonymized when redacting) report so a redacted file never leaks the real id and is never name-identical to a raw export.</p><p>A finding row carries <code>impactEstimate</code> when the finding has one, with the full contract documented in <a href="./../impact-estimation.html#occupancy-weighted-attribution">Impact estimation</a> (basis, wallClock, estimateMethod, rawWaste, coreTimeMs, and on <code>utilization</code> idleCoreTimeMs). It appears after the pinned core columns (<code>id</code>, <code>type</code>, <code>impactBand</code>, <code>stageId</code>, <code>metric</code>, <code>value</code>, <code>recommendation</code>, <code>detectorVersion</code>, plus optional but pinned <code>confidence</code>, <code>validationRequired</code>, <code>docAnchor</code>) without displacing any of them. The full row shape is <code>FindingRowColumns</code> in <code>evidence-report.ts</code>, which also carries <code>name</code>, <code>tag</code>, <code>valueText</code>, <code>remediation</code>, <code>actionLabel</code>, <code>impact</code>/<code>impactMeaning</code> and, on a tuned run, <code>tunedThresholds</code>.</p><p>The report carries the same recommendation data the dashboard&#39;s &quot;Fix These First&quot; view shows, so the CLI (<code>packages/cli/bin/sparkforensics-analyze.mjs</code>) and the MCP tool (<code>diagnoseRun</code> in <code>packages/core/src/mcp-tools.ts</code>) get it too, not just the web markdown/JSON download. <code>FindingRow.actionLabel</code> is always present (a short imperative label like &quot;Reduce shuffle size&quot;), from <code>findingActionLabel</code> (<code>packages/core/src/finding-action-label.ts</code>), which reads the type&#39;s <code>actionLabel</code> in <code>finding-presentation.ts</code> and falls back to the type&#39;s name. The dashboard, the run verdict and the report rows all call it. <code>EvidenceReportJson.recommendations</code> is the same impact-ranked <code>buildRecommendationRollup</code> grouping (<code>packages/core/src/recommendation-rollup.ts</code>) that <code>FixTheseFirst.tsx</code> renders, so CLI/MCP/download consumers get the same &quot;what&#39;s the highest-impact fix&quot; ranking the dashboard shows, without changing the <code>findings</code> array&#39;s own impact-band-sorted order at all. <code>EvidenceReportJson.cleanChecks</code> lists every detector type that fired zero findings this run, each with <code>getThresholdSummary</code>&#39;s one-line &quot;what would have tripped it&quot; sentence; unlike the dashboard&#39;s <code>Alerts.tsx</code> clean-checks table, it deliberately includes the one &quot;always-mounted&quot; reference type (<code>coreLocality</code>) even when it has no findings, since a flat evidence report has no separate always-visible surface for it to already appear on the way that widget does on the board.</p><p>A check the log could not run is not listed as clean. <code>packages/core/src/check-coverage.ts</code> holds the one rule, shared with the dashboard&#39;s verdict, top bar and Clean checks: a type whose only finding is an evidence caveat, every <code>scope: &#39;stage&#39;</code> type on a log where no stage recorded an end, and the run-span types (<code>utilization</code>, <code>memoryUtilization</code>, <code>autoscalingChurn</code>) on a log with no ApplicationEnd. Those types are listed in <code>notRunChecks</code> instead of <code>cleanChecks</code>, each <code>{ type, tag, thresholdSummary, reason }</code>, where <code>reason</code> is the caveat&#39;s own recommendation (it names the setting to turn on) or the log-wide sentence. The Markdown has a <code>## Not checked on this log</code> section above <code>## Clean checks</code>, and a <code>Findings to act on</code> header line.</p><p><code>Finding</code> is a union discriminated on <code>type</code>, one member per emitted finding type (<code>packages/core/src/finding-types.ts</code>), and a finding row&#39;s <code>evidence</code> is an explicit per-type projection. Each type&#39;s <code>&lt;Type&gt;Evidence</code> interface names its public fields, and <code>EVIDENCE_KEYS</code> in <code>evidence-report.ts</code> lists the same keys, checked both ways at compile time, so a field a detector adds only for another core module does not become report contract. Rows leave out these fields: <code>stageShape</code>&#39;s <code>totalCores</code>; <code>utilization</code>&#39;s <code>utilizationFraction</code>, <code>appDurationMs</code>, <code>totalCores</code> and <code>allocatedCoreMs</code>; <code>memoryUtilization</code>&#39;s <code>idleRateFraction</code>, <code>allocatedMBSeconds</code>, <code>allocatedMB</code>, <code>peakExecutors</code>, <code>appDurationMs</code> and <code>allocatedBytes</code>; <code>retryWaste</code>&#39;s <code>extended</code> display copy. The impact estimator reads them on the finding. <code>value</code> is always numeric or <code>null</code>: the text-valued findings (<code>stageFailed</code>&#39;s failure reason, <code>configAudit</code>&#39;s current setting, <code>incompleteRun</code>&#39;s <code>missing</code>) carry their text in a <code>valueText</code> column, present only on those rows, and the Markdown prints it where <code>value</code> would go. Renaming or removing an evidence field is a breaking change and needs a schema bump. <code>cleanChecks</code>/<code>notRunChecks</code> list emitted finding types, the set the dashboard&#39;s Clean checks shows: <code>overBroadcast</code> and <code>underBroadcast</code> in place of the <code>broadcastSizing</code> detector entry. A finding row&#39;s <code>actionLabel</code> for a (type, discriminant) combination with no label of its own is the type&#39;s name, the same fallback the verdict step uses.</p><p><code>buildEvidenceReport(appModel, { thresholds })</code> runs the detectors with a user&#39;s validated overrides (the CLI&#39;s and MCP server&#39;s <code>--thresholds</code>; see <a href="./../detector-contract.html#tuning-thresholds">Tuning thresholds</a>). Only a tuned run adds keys: <code>tunedThresholds</code> (<code>{ &lt;name&gt;: { value, default } }</code>) on each finding row and each <code>cleanChecks</code>/<code>notRunChecks</code> entry from a detector an override moved off its defaults (or whose <code>suppressedBy</code> detector it moved), on that detector&#39;s <code>detectors</code> row, and as <code>summary.tunedThresholds</code> keyed by detector type. The Markdown adds a <code>- Tuned thresholds:</code> header line, a <code>- tuned thresholds:</code> line per affected finding, and marks tuned catalog rows and clean checks. A default run&#39;s report carries none of these keys, and every one is optional. The report caches key on the overrides object as well as the <code>appModel</code>, so one model&#39;s default and tuned reports never mix. Finding ids ignore thresholds: a finding a tuned run still emits keeps the id it has in a default run.</p><h2 id="finding-identity" tabindex="-1">Finding identity <a class="header-anchor" href="#finding-identity" aria-label="Permalink to &quot;Finding identity&quot;">​</a></h2><p>Every finding <code>analyzer.ts</code> emits carries a stable <code>id</code> (<code>push()</code>&#39;s <code>findingId()</code> choke point, FNV-1a hash over <code>type | location | metric | value | discriminators</code>) and a <code>detectorVersion</code> (from the emitting <code>DETECTORS</code> entry). The two are deliberately decoupled: <code>id</code> is derived only from a finding&#39;s evidence tuple, never from <code>detectorVersion</code>, so a threshold or logic tweak that bumps a detector&#39;s <code>version</code> does not change the <code>id</code> of findings it still emits at the same location/metric. A saved reference (dashboard bookmark, exported report row) keeps pointing at the same logical finding across detector revisions; <code>detectorVersion</code> is separate provenance metadata for &quot;which ruleset produced this,&quot; not part of identity. See <code>packages/core/test/analyzer-finding-identity.test.js</code> for the decoupling proof.</p><p>The discriminators are declared per finding type (<code>ID_DISCRIMINATORS</code> in <code>analyzer.ts</code>, each key checked against that type&#39;s fields) and joined in the fixed <code>DISCRIMINATOR_SLOTS</code> order, and the value slot takes <code>value</code>, else <code>valueText</code>.</p><p><code>id</code> stability holds for equivalent reruns of the same schema/analyzer version on the same input. Changing the id-derivation rule itself (the hash algorithm, or which fields feed the location key/discriminators) is an intentional breaking change and must bump <code>EVIDENCE_SCHEMA_VERSION</code> (<code>packages/core/src/evidence-report.ts</code>); there is no separate id-scheme version.</p>',19)])])}const m=o(a,[["render",n]]);export{u as __pageData,m as default};

package/public/docs/assets/{contributor-guide_development-setup.md.D9bu4j_c.js → contributor-guide_development-setup.md.DQdzera7.js}

@@ -12,4 +12,4 @@ import{_ as s,o as t,c as i,a5 as a}from"./chunks/framework.DSg0KOwT.js";const k
 <span class="line"><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">npm</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> run</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> lint</span><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;">          # eslint over the repo&#39;s JS (.ts/.tsx are covered only by tsc)</span></span>
 <span class="line"><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">node</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> packages/cli/bin/sparkforensics-analyze.mjs</span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;"> &lt;</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">file</span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">|</span><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">rolling-log-dir</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">&gt;   </span><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># run the CLI analyzer against a log, outside the browser</span></span></code></pre></div><p><code>npm run test:coverage</code>, at the root or inside <code>packages/core</code>, <code>cli</code>, <code>mcp</code> or <code>server</code>, runs that suite with <code>--coverage</code> (v8 provider, <code>lcov</code> and <code>text</code> reporters, plus <code>html</code> at the root; written to <code>&lt;pkg&gt;/coverage/lcov.info</code>). CI&#39;s <code>build</code>, <code>core</code>, <code>cli</code>, <code>mcp</code> and <code>server</code> jobs all run through it and upload to Coveralls, merged by a final <code>finish</code> job.</p><p>The dashboard&#39;s <strong>Download HTML dashboard</strong> fetches <code>export-template.html</code> from beside the app, so under <code>npm run dev</code> it fails with an error toast; use <code>npm run build &amp;&amp; npm run preview</code> to try it.</p><p><code>npm install</code> also runs the <code>prepare</code> script, which points git at <code>.githooks/</code>. After that, a pre-commit hook lints staged JS files on every commit. Fix what it reports, or skip it with <code>git commit --no-verify</code>.</p><p>The tuning reference under <code>packages/core/src/docs-content/{chapters,tuning,diagrams}</code> isn&#39;t committed. It&#39;s generated from the <code>shuffle-works/spark-tuning-reference</code> commit pinned in <code>packages/core/src/docs-content/upstream.json</code>: the first <code>npm test</code>, <code>npm run docs:dev</code> or <code>npm run build</code> fetches it over HTTPS (about 2s, no credentials) and later runs reuse the gitignored copy while it matches the pin. <code>npm run docs:fetch</code> does the same step on its own. Never edit those folders: fix the content upstream, then <code>npm run docs:bump</code> to move the pin. Running the MCP or CLI bins from source (<code>node packages/*/bin/...</code>) doesn&#39;t fetch it, so run <code>npm run docs:fetch</code> first: without it the MCP&#39;s tuning-reference lookups come back empty. <code>npm run docs:bump [-- &lt;sha&gt;]</code> moves the pin (default: upstream head), regenerates the docs and writes the changeset; put the compare link it prints in the PR body. <code>npm run docs:bump -- --check [&lt;sha&gt;]</code> runs the anchor gate and writes nothing; <code>.github/workflows/tuning-reference-drift.yml</code> runs it with <code>test:core</code> every week.</p><p>Offline, a cached copy keeps working. If the pin has moved since, <code>npm test</code> and <code>docs:dev</code> warn and use the older copy, while <code>docs:build</code>, <code>npm pack</code> and anything with <code>CI</code> set fail. To build from a local upstream checkout instead (offline, or to preview upstream edits in this docs site before they merge), set <code>SPARK_TUNING_REFERENCE_DIR</code>:</p><div class="language-bash vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">bash</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">SPARK_TUNING_REFERENCE_DIR</span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">=</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">../spark-tuning-reference</span><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;"> npm</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> run</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> docs:dev</span></span></code></pre></div><p>The same checks apply to that checkout. <code>npm pack</code> and CI accept it only when it sits at the pinned commit with no uncommitted <code>content/</code> changes.</p><p><code>dev/log-corpus</code> is a git submodule pointing at the public <code>spark-event-corpus-data</code> repo. It&#39;s optional locally: if you skip the <code>git submodule update</code> step above, the corpus-backed tests (such as <code>packages/server/test/shs-proxy-fixture.test.js</code>) report as skipped rather than failed, which is expected, not a bug. CI&#39;s <code>core</code> and <code>server</code> jobs check it out and run them (plus the <a href="./testing.html#corpus-regression-snapshot">corpus regression snapshot</a> in <code>core</code>); the <code>build</code>, <code>cli</code> and <code>mcp</code> jobs don&#39;t, so a corpus-backed test belongs in <code>packages/core</code> or <code>packages/server</code>.</p><p>The <code>packages/server/</code> package (the optional local-server deploy mode) keeps its own dependencies and test suite. Run it from the repo root: a second <code>npm install</code> inside <code>packages/server</code> prunes the shared root <code>node_modules</code>&#39;s workspace symlinks.</p><div class="language-bash vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">bash</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">npm</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> run</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> test:server</span><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;">   # runs packages/server&#39;s own suite, not part of \`npm test\`</span></span></code></pre></div><p>Packing or releasing <code>packages/server/</code> builds the docs site for you: its <code>prepack</code> runs <code>packages/server/scripts/copy-frontend.js</code>, which runs the root <code>npm run build</code> (<code>docs:build</code> included), so the packed artifact always ships <code>/docs/</code>.</p><p>This docs site has its own dev loop:</p><div class="language-bash vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">bash</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">npm</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> run</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> docs:dev</span><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;">       # VitePress dev server for docs-site/</span></span>
 <span class="line"><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">npm</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> run</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> docs:build</span><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;">     # production build; also checks internal links/anchors</span></span>
-<span class="line"><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">npm</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> run</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> docs:preview</span><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;">   # serves the built docs site locally</span></span></code></pre></div><h2 id="generated-and-vendored-files" tabindex="-1">Generated and vendored files <a class="header-anchor" href="#generated-and-vendored-files" aria-label="Permalink to &quot;Generated and vendored files&quot;">​</a></h2><ul><li><code>packages/core/src/docs-content/detection/*.md</code> is generated from <code>docs-site/user-guide/understanding-findings.md</code>. After editing that guide, run <code>npm run split-detection-docs</code> and commit the output, or <code>tests/detection-docs-split.test.js</code> fails.</li><li><code>packages/{cli,mcp,server}/vendor-core/</code> (gitignored) is rebuilt only at <code>prepack</code>. In the monorepo the bins load <code>packages/core/src/load-vendored.js</code>, which uses a leftover <code>vendor-core/</code> only while its <code>core-source-hash.txt</code> matches <code>packages/core/src</code>; otherwise it warns on stderr and runs the source.</li><li>The CLI&#39;s <code>export-template/</code> has no such check. Without it, <code>--export-html</code> from source uses the repo-root <code>dist-export/</code> from <code>npm run build:export</code> (not produced by <code>npm run build</code>); a leftover <code>export-template/</code> wins over that, so rebuild it with <code>node scripts/vendor-export-template.mjs</code> or delete it before trusting a local <code>--export-html</code>.</li></ul><h2 id="the-docs-site-s-anchor-clicks" tabindex="-1">The docs site&#39;s anchor clicks <a class="header-anchor" href="#the-docs-site-s-anchor-clicks" aria-label="Permalink to &quot;The docs site&#39;s anchor clicks&quot;">​</a></h2><p>VitePress installs a <code>window</code>-level, capture-phase click listener (<code>node_modules/vitepress/dist/client/app/router.js</code>) that catches clicks on any <code>&lt;a href=&quot;#...&quot;&gt;</code> pointing at the current page and scrolls to the target itself, before a handler the theme adds on that element runs, so <code>preventDefault()</code> there is too late. An interactive control built over in-page anchors, such as the footnote-marker popovers in <code>docs-site/.vitepress/theme/citation-chips.ts</code>, must drop the element&#39;s <code>href</code> and restore <code>role</code>, <code>tabindex</code> and keyboard handling by hand.</p><p>That router scrolls with its own offset, while a deep link, and every click on the published tuning reference (which ships without hydration), is a native fragment jump that follows <code>scroll-padding-top</code>. Both read one measured value: <code>docs-site/.vitepress/anchor-offset.js</code>, inlined into every page&#39;s head, sizes it from the pinned chrome on screen, which includes the Shuffle Works product bar on the published site. Never write that bar&#39;s marker attribute literally in the docs source, prose included: the hub skips injecting its bar into any built page that contains it.</p><h2 id="driving-the-app-in-a-browser" tabindex="-1">Driving the app in a browser <a class="header-anchor" href="#driving-the-app-in-a-browser" aria-label="Permalink to &quot;Driving the app in a browser&quot;">​</a></h2><ul><li>Sample logs: download one from a Spark History Server with <code>curl -o app.zip &quot;&lt;baseUrl&gt;/api/v1/applications/&lt;appId&gt;/logs&quot;</code> and drop the zip in as-is (single-file and rolling logs both unwrap), or the <code>.zstd</code> event log inside it. Private logs follow the naming rule in <a href="./testing.html">Testing</a>.</li><li>Loading a file under Playwright: the drop zone prefers <code>showOpenFilePicker</code>, a native dialog that localhost Chromium can&#39;t drive. Run <code>await page.evaluate(() =&gt; delete window.showOpenFilePicker)</code> before clicking <strong>Choose file</strong>, then <code>setInputFiles</code> on <code>[data-testid=file-input]</code> or handle the <code>filechooser</code> event.</li><li>Run comparison: on the landing page, click <strong>Compare two runs</strong>, fill the <strong>Baseline</strong> and <strong>Candidate</strong> slots the same way (each has its own <code>[data-testid=file-input]</code>), then click <strong>Compare</strong>. Both runs parse one after the other through the one worker. From a run&#39;s dashboard, the topbar&#39;s <strong>Compare with another run</strong> opens the same view with that run in the baseline slot (a <code>cached</code> <code>RunSource</code>, not parsed again). <strong>View baseline/candidate dashboard</strong> drills into one run, and <strong>← Back to comparison</strong> returns.</li><li>The landing page&#39;s <strong>Try a sample run</strong> (<code>src/view/DropZone.tsx</code>) loads a gzip-compressed corpus log from <code>public/sample-runs/</code>: the corpus candidate with the most findings under <code>sparkforensics-analyze --format json</code>. Rerun that scan over every candidate, against the current corpus, before swapping the bundled sample.</li><li>Screenshots on a PR: the <code>gh</code> token can&#39;t use GitHub&#39;s browser-only attachment uploader. Commit the PNGs to a throwaway asset branch and embed them by commit SHA as <code>https://raw.githubusercontent.com/&lt;owner&gt;/&lt;repo&gt;/&lt;sha&gt;/&lt;path&gt;.png</code>. The repo is public, so every reader&#39;s browser loads them, and the binaries stay out of the feature diff.</li></ul>`,24)])])}const u=s(o,[["render",n]]);export{k as __pageData,u as default};
+<span class="line"><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">npm</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> run</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> docs:preview</span><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;">   # serves the built docs site locally</span></span></code></pre></div><h2 id="generated-and-vendored-files" tabindex="-1">Generated and vendored files <a class="header-anchor" href="#generated-and-vendored-files" aria-label="Permalink to &quot;Generated and vendored files&quot;">​</a></h2><ul><li><code>packages/core/src/docs-content/detection/*.md</code> (gitignored) is split from <code>docs-site/user-guide/understanding-findings.md</code>, one file per finding tag. <code>docs:dev</code>, <code>docs:build</code>, the vitest globalSetup and <code>prepack</code> regenerate it when it differs from the guide, so nothing is committed or run by hand. To regenerate on demand: <code>npm run split-detection-docs</code>.</li><li><code>packages/{cli,mcp,server}/vendor-core/</code> (gitignored) is rebuilt only at <code>prepack</code>. In the monorepo the bins load <code>packages/core/src/load-vendored.js</code>, which uses a leftover <code>vendor-core/</code> only while its <code>core-source-hash.txt</code> matches <code>packages/core/src</code>; otherwise it warns on stderr and runs the source.</li><li>The CLI&#39;s <code>export-template/</code> has no such check. Without it, <code>--export-html</code> from source uses the repo-root <code>dist-export/</code> from <code>npm run build:export</code> (not produced by <code>npm run build</code>); a leftover <code>export-template/</code> wins over that, so rebuild it with <code>node scripts/vendor-export-template.mjs</code> or delete it before trusting a local <code>--export-html</code>.</li></ul><h2 id="the-docs-site-s-anchor-clicks" tabindex="-1">The docs site&#39;s anchor clicks <a class="header-anchor" href="#the-docs-site-s-anchor-clicks" aria-label="Permalink to &quot;The docs site&#39;s anchor clicks&quot;">​</a></h2><p>VitePress installs a <code>window</code>-level, capture-phase click listener (<code>node_modules/vitepress/dist/client/app/router.js</code>) that catches clicks on any <code>&lt;a href=&quot;#...&quot;&gt;</code> pointing at the current page and scrolls to the target itself, before a handler the theme adds on that element runs, so <code>preventDefault()</code> there is too late. An interactive control built over in-page anchors, such as the footnote-marker popovers in <code>docs-site/.vitepress/theme/citation-chips.ts</code>, must drop the element&#39;s <code>href</code> and restore <code>role</code>, <code>tabindex</code> and keyboard handling by hand.</p><p>That router scrolls with its own offset, while a deep link, and every click on the published tuning reference (which ships without hydration), is a native fragment jump that follows <code>scroll-padding-top</code>. Both read one measured value: <code>docs-site/.vitepress/anchor-offset.js</code>, inlined into every page&#39;s head, sizes it from the pinned chrome on screen, which includes the Shuffle Works product bar on the published site. Never write that bar&#39;s marker attribute literally in the docs source, prose included: the hub skips injecting its bar into any built page that contains it.</p><h2 id="driving-the-app-in-a-browser" tabindex="-1">Driving the app in a browser <a class="header-anchor" href="#driving-the-app-in-a-browser" aria-label="Permalink to &quot;Driving the app in a browser&quot;">​</a></h2><ul><li>Sample logs: download one from a Spark History Server with <code>curl -o app.zip &quot;&lt;baseUrl&gt;/api/v1/applications/&lt;appId&gt;/logs&quot;</code> and drop the zip in as-is (single-file and rolling logs both unwrap), or the <code>.zstd</code> event log inside it. Private logs follow the naming rule in <a href="./testing.html">Testing</a>.</li><li>Loading a file under Playwright: the drop zone prefers <code>showOpenFilePicker</code>, a native dialog that localhost Chromium can&#39;t drive. Run <code>await page.evaluate(() =&gt; delete window.showOpenFilePicker)</code> before clicking <strong>Choose file</strong>, then <code>setInputFiles</code> on <code>[data-testid=file-input]</code> or handle the <code>filechooser</code> event.</li><li>Run comparison: on the landing page, click <strong>Compare two runs</strong>, fill the <strong>Baseline</strong> and <strong>Candidate</strong> slots the same way (each has its own <code>[data-testid=file-input]</code>), then click <strong>Compare</strong>. Both runs parse one after the other through the one worker. From a run&#39;s dashboard, the topbar&#39;s <strong>Compare with another run</strong> opens the same view with that run in the baseline slot (a <code>cached</code> <code>RunSource</code>, not parsed again). <strong>View baseline/candidate dashboard</strong> drills into one run, and <strong>← Back to comparison</strong> returns.</li><li>The landing page&#39;s <strong>Try a sample run</strong> (<code>src/view/DropZone.tsx</code>) loads a gzip-compressed corpus log from <code>public/sample-runs/</code>: the corpus candidate with the most findings under <code>sparkforensics-analyze --format json</code>. Rerun that scan over every candidate, against the current corpus, before swapping the bundled sample.</li><li>Screenshots on a PR: the <code>gh</code> token can&#39;t use GitHub&#39;s browser-only attachment uploader. Commit the PNGs to a throwaway asset branch and embed them by commit SHA as <code>https://raw.githubusercontent.com/&lt;owner&gt;/&lt;repo&gt;/&lt;sha&gt;/&lt;path&gt;.png</code>. The repo is public, so every reader&#39;s browser loads them, and the binaries stay out of the feature diff.</li></ul>`,24)])])}const u=s(o,[["render",n]]);export{k as __pageData,u as default};