sparkforensics-cli 0.2.0 → 0.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/export-template/docs/404.html +1 -1
- package/export-template/docs/assets/{app.CndaAS6v.js → app.DQTZyGL1.js} +1 -1
- package/export-template/docs/assets/chunks/@localSearchIndexroot.DppnXnDE.js +1 -0
- package/export-template/docs/assets/chunks/{VPLocalSearchBox.yJbZbsEo.js → VPLocalSearchBox.BkBIPFs6.js} +1 -1
- package/export-template/docs/assets/chunks/theme.DP0u1AUq.js +2 -0
- package/export-template/docs/assets/contributor-guide_architecture_board-widgets.md.CWpj01WU.js +1 -0
- package/export-template/docs/assets/contributor-guide_architecture_board-widgets.md.CWpj01WU.lean.js +1 -0
- package/export-template/docs/assets/contributor-guide_architecture_detector-contract.md.CgzUsQ6W.js +1 -0
- package/export-template/docs/assets/{contributor-guide_architecture_impact-estimation.md.DYCDPgkh.js → contributor-guide_architecture_impact-estimation.md.CooslVJt.js} +1 -1
- package/export-template/docs/assets/contributor-guide_architecture_state-and-history.md.C-xxn0q7.js +1 -0
- package/export-template/docs/assets/contributor-guide_architecture_state-and-history.md.C-xxn0q7.lean.js +1 -0
- package/export-template/docs/assets/contributor-guide_architecture_widget-rendering.md.R27gQrgY.js +1 -0
- package/export-template/docs/assets/contributor-guide_architecture_worker-protocol.md.IbnfNrV3.js +6 -0
- package/export-template/docs/assets/{style.DXOMCXxn.css → style.DSixAiZE.css} +1 -1
- package/export-template/docs/assets/{user-guide_alternative-log-retrieval.md.sU3KGarf.js → user-guide_alternative-log-retrieval.md.B4tPGIal.js} +1 -1
- package/export-template/docs/assets/{user-guide_alternative-log-retrieval.md.sU3KGarf.lean.js → user-guide_alternative-log-retrieval.md.B4tPGIal.lean.js} +1 -1
- package/export-template/docs/assets/user-guide_getting-started.md.BJvwLEIM.js +3 -0
- package/export-template/docs/assets/user-guide_getting-started.md.BJvwLEIM.lean.js +1 -0
- package/export-template/docs/assets/{user-guide_mcp-tools.md.C8MiIu7F.js → user-guide_mcp-tools.md.Vi3RoflJ.js} +3 -3
- package/export-template/docs/assets/{user-guide_mcp-tools.md.C8MiIu7F.lean.js → user-guide_mcp-tools.md.Vi3RoflJ.lean.js} +1 -1
- package/export-template/docs/assets/user-guide_run-comparison.md.CQc1aoU8.js +1 -0
- package/export-template/docs/assets/user-guide_run-comparison.md.CQc1aoU8.lean.js +1 -0
- package/export-template/docs/assets/user-guide_understanding-findings.md.DL1UDhvR.js +1 -0
- package/export-template/docs/assets/user-guide_understanding-findings.md.DL1UDhvR.lean.js +1 -0
- package/export-template/docs/contributor-guide/architecture/board-widgets.html +2 -2
- package/export-template/docs/contributor-guide/architecture/detector-contract.html +2 -2
- package/export-template/docs/contributor-guide/architecture/drill-down.html +1 -1
- package/export-template/docs/contributor-guide/architecture/impact-estimation.html +2 -2
- package/export-template/docs/contributor-guide/architecture/index.html +1 -1
- package/export-template/docs/contributor-guide/architecture/overview.html +1 -1
- package/export-template/docs/contributor-guide/architecture/state-and-history.html +2 -2
- package/export-template/docs/contributor-guide/architecture/widget-rendering.html +2 -2
- package/export-template/docs/contributor-guide/architecture/worker-protocol.html +2 -2
- package/export-template/docs/contributor-guide/contributing.html +1 -1
- package/export-template/docs/contributor-guide/development-setup.html +1 -1
- package/export-template/docs/contributor-guide/testing.html +1 -1
- package/export-template/docs/index.html +1 -1
- package/export-template/docs/tuning-reference/anti-patterns.html +1 -1
- package/export-template/docs/tuning-reference/aqe.html +1 -1
- package/export-template/docs/tuning-reference/bottleneck-broadcast-sizing.html +1 -1
- package/export-template/docs/tuning-reference/bottleneck-cold-start.html +1 -1
- package/export-template/docs/tuning-reference/bottleneck-duplicate-plan-subtree.html +1 -1
- package/export-template/docs/tuning-reference/bottleneck-failures.html +1 -1
- package/export-template/docs/tuning-reference/bottleneck-gc.html +1 -1
- package/export-template/docs/tuning-reference/bottleneck-job-failure-rate.html +1 -1
- package/export-template/docs/tuning-reference/bottleneck-memory-utilization.html +1 -1
- package/export-template/docs/tuning-reference/bottleneck-retry-waste.html +1 -1
- package/export-template/docs/tuning-reference/bottleneck-shuffle.html +1 -1
- package/export-template/docs/tuning-reference/bottleneck-skew.html +1 -1
- package/export-template/docs/tuning-reference/bottleneck-slow-host.html +1 -1
- package/export-template/docs/tuning-reference/bottleneck-small-files.html +1 -1
- package/export-template/docs/tuning-reference/bottleneck-spill.html +1 -1
- package/export-template/docs/tuning-reference/bottleneck-straggler.html +1 -1
- package/export-template/docs/tuning-reference/bottleneck-tiny-tasks.html +1 -1
- package/export-template/docs/tuning-reference/bottleneck-utilization.html +1 -1
- package/export-template/docs/tuning-reference/caching.html +1 -1
- package/export-template/docs/tuning-reference/cluster-config.html +1 -1
- package/export-template/docs/tuning-reference/config.html +1 -1
- package/export-template/docs/tuning-reference/data-formats.html +1 -1
- package/export-template/docs/tuning-reference/index.html +1 -1
- package/export-template/docs/tuning-reference/intro.html +1 -1
- package/export-template/docs/tuning-reference/joins.html +1 -1
- package/export-template/docs/tuning-reference/memory-model.html +1 -1
- package/export-template/docs/tuning-reference/metrics.html +1 -1
- package/export-template/docs/tuning-reference/partitioning.html +1 -1
- package/export-template/docs/tuning-reference/pyspark.html +1 -1
- package/export-template/docs/tuning-reference/shuffle.html +1 -1
- package/export-template/docs/tuning-reference/spark-architecture.html +1 -1
- package/export-template/docs/tuning-reference/table-formats.html +1 -1
- package/export-template/docs/user-guide/alternative-log-retrieval.html +2 -2
- package/export-template/docs/user-guide/getting-started.html +3 -3
- package/export-template/docs/user-guide/mcp-tools.html +4 -4
- package/export-template/docs/user-guide/run-comparison.html +2 -2
- package/export-template/docs/user-guide/understanding-findings.html +2 -2
- package/export-template/index.html +75 -79
- package/export-template/parser-worker-DyjiQvfP.js +112 -0
- package/export-template/sample-runs/sample-run.ndjson.gz +0 -0
- package/package.json +5 -4
- package/vendor-core/detectors.js +149 -24
- package/vendor-core/docs-content/detection/cache.md +4 -3
- package/vendor-core/docs-content/detection/chrn.md +4 -2
- package/vendor-core/docs-content/detection/local.md +2 -3
- package/vendor-core/docs-content/detection/mem.md +3 -3
- package/vendor-core/docs-content/detection/spec.md +4 -3
- package/vendor-core/evidence-report.js +1 -1
- package/vendor-core/parser-worker.js +2 -2
- package/vendor-core/run-comparison.js +21 -2
- package/export-template/docs/assets/chunks/@localSearchIndexroot.DNY8bVcl.js +0 -1
- package/export-template/docs/assets/chunks/theme.Df2VAG9w.js +0 -2
- package/export-template/docs/assets/contributor-guide_architecture_board-widgets.md.B-OsL91z.js +0 -1
- package/export-template/docs/assets/contributor-guide_architecture_board-widgets.md.B-OsL91z.lean.js +0 -1
- package/export-template/docs/assets/contributor-guide_architecture_detector-contract.md.BOeH4d1J.js +0 -1
- package/export-template/docs/assets/contributor-guide_architecture_state-and-history.md.m3S3UdMk.js +0 -1
- package/export-template/docs/assets/contributor-guide_architecture_state-and-history.md.m3S3UdMk.lean.js +0 -1
- package/export-template/docs/assets/contributor-guide_architecture_widget-rendering.md.DbqPf2OT.js +0 -1
- package/export-template/docs/assets/contributor-guide_architecture_worker-protocol.md.B93qJ_tT.js +0 -6
- package/export-template/docs/assets/user-guide_getting-started.md.DtEM37MK.js +0 -3
- package/export-template/docs/assets/user-guide_getting-started.md.DtEM37MK.lean.js +0 -1
- package/export-template/docs/assets/user-guide_run-comparison.md.S0TWWmLY.js +0 -1
- package/export-template/docs/assets/user-guide_run-comparison.md.S0TWWmLY.lean.js +0 -1
- package/export-template/docs/assets/user-guide_understanding-findings.md.D0R_Y-R2.js +0 -1
- package/export-template/docs/assets/user-guide_understanding-findings.md.D0R_Y-R2.lean.js +0 -1
- package/export-template/parser-worker-QqyEE4m9.js +0 -64
- /package/export-template/docs/assets/{contributor-guide_architecture_detector-contract.md.BOeH4d1J.lean.js → contributor-guide_architecture_detector-contract.md.CgzUsQ6W.lean.js} +0 -0
- /package/export-template/docs/assets/{contributor-guide_architecture_impact-estimation.md.DYCDPgkh.lean.js → contributor-guide_architecture_impact-estimation.md.CooslVJt.lean.js} +0 -0
- /package/export-template/docs/assets/{contributor-guide_architecture_widget-rendering.md.DbqPf2OT.lean.js → contributor-guide_architecture_widget-rendering.md.R27gQrgY.lean.js} +0 -0
- /package/export-template/docs/assets/{contributor-guide_architecture_worker-protocol.md.B93qJ_tT.lean.js → contributor-guide_architecture_worker-protocol.md.IbnfNrV3.lean.js} +0 -0
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="./assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="./assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="./vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Anti-Patterns | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Adaptive Query Execution | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Broadcast Sizing | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Cold Start | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Duplicate Plan Subtree | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Task Failures | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>GC Pressure | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Job Failure Rate | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Memory Utilization | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Retry Waste | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Shuffle I/O | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Task Skew | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Slow Host | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Small Files | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Memory / Disk Spill | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Stragglers | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Tiny Tasks | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Executor Utilization | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Caching & Persistence | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Cluster Tuning | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Spark Config Quick-Reference | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Data Formats | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Spark Tuning Reference | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Introduction | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Join Optimization | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Memory Management | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Metrics Glossary | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Partitioning | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>PySpark Specifics | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Shuffle | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Spark Execution Model | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Table Formats | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Alternative ways to get the logs | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -18,7 +18,7 @@
|
|
|
18
18
|
<script id="check-mac-os">document.documentElement.classList.toggle("mac",/Mac|iPhone|iPod|iPad/i.test(navigator.platform));</script>
|
|
19
19
|
</head>
|
|
20
20
|
<body>
|
|
21
|
-
<div id="app"><div class="Layout" data-v-5d98c3a5><!--[--><!--]--><!--[--><span tabindex="-1" data-v-0b0ada53></span><a href="#VPContent" class="VPSkipLink visually-hidden" data-v-0b0ada53>Skip to content</a><!--]--><!----><header class="VPNav" data-v-5d98c3a5 data-v-ae24b3ad><div class="VPNavBar" data-v-ae24b3ad data-v-6aa21345><div class="wrapper" data-v-6aa21345><div class="container" data-v-6aa21345><div class="title" data-v-6aa21345><div class="VPNavBarTitle has-sidebar" data-v-6aa21345 data-v-1168a8e4><a class="title" href="../index.html" data-v-1168a8e4><!--[--><!--]--><!--[--><img class="VPImage logo" src="../favicon.svg" alt data-v-8426fc1a><!--]--><span data-v-1168a8e4>SparkForensics</span><!--[--><!--]--></a></div></div><div class="content" data-v-6aa21345><div class="content-body" data-v-6aa21345><!--[--><!--]--><div class="VPNavBarSearch search" data-v-6aa21345><!--[--><!----><div id="local-search"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><span class="vp-icon DocSearch-Search-Icon"></span><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"><kbd class="DocSearch-Button-Key"></kbd><kbd class="DocSearch-Button-Key">K</kbd></span></button></div><!--]--></div><nav aria-labelledby="main-nav-aria-label" class="VPNavBarMenu menu" data-v-6aa21345 data-v-dc692963><span id="main-nav-aria-label" class="visually-hidden" data-v-dc692963> Main Navigation </span><!--[--><!--[--><a class="VPLink link VPNavBarMenuLink" href="../user-guide/getting-started.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>User Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="../contributor-guide/development-setup.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Contributor Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="../tuning-reference/index.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Tuning Reference</span><!--]--></a><!--]--><!--]--></nav><!----><div class="VPNavBarAppearance appearance" data-v-6aa21345 data-v-6c893767><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-6c893767 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div><!----><div class="VPFlyout VPNavBarExtra extra" data-v-6aa21345 data-v-bb2aa2f0 data-v-cf11d7a2><button type="button" class="button" aria-haspopup="true" aria-expanded="false" aria-label="extra navigation" data-v-cf11d7a2><span class="vpi-more-horizontal icon" data-v-cf11d7a2></span></button><div class="menu" data-v-cf11d7a2><div class="VPMenu" data-v-cf11d7a2 data-v-b98bc113><!----><!--[--><!--[--><!----><div class="group" data-v-bb2aa2f0><div class="item appearance" data-v-bb2aa2f0><p class="label" data-v-bb2aa2f0>Appearance</p><div class="appearance-action" data-v-bb2aa2f0><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-bb2aa2f0 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div></div></div><!----><!--]--><!--]--></div></div></div><!--[--><!--]--><button type="button" class="VPNavBarHamburger hamburger" aria-label="mobile navigation" aria-expanded="false" aria-controls="VPNavScreen" data-v-6aa21345 data-v-e5dd9c1c><span class="container" data-v-e5dd9c1c><span class="top" data-v-e5dd9c1c></span><span class="middle" data-v-e5dd9c1c></span><span class="bottom" data-v-e5dd9c1c></span></span></button></div></div></div></div><div class="divider" data-v-6aa21345><div class="divider-line" data-v-6aa21345></div></div></div><!----></header><div class="VPLocalNav has-sidebar empty" data-v-5d98c3a5 data-v-a6f0e41e><div class="container" data-v-a6f0e41e><button class="menu" aria-expanded="false" aria-controls="VPSidebarNav" data-v-a6f0e41e><span class="vpi-align-left menu-icon" data-v-a6f0e41e></span><span class="menu-text" data-v-a6f0e41e>Menu</span></button><div class="VPLocalNavOutlineDropdown" style="--vp-vh:0px;" data-v-a6f0e41e data-v-8a42e2b4><button data-v-8a42e2b4>Return to top</button><!----></div></div></div><aside class="VPSidebar" data-v-5d98c3a5 data-v-319d5ca6><div class="curtain" data-v-319d5ca6></div><nav class="nav" id="VPSidebarNav" aria-labelledby="sidebar-aria-label" tabindex="-1" data-v-319d5ca6><span class="visually-hidden" id="sidebar-aria-label" data-v-319d5ca6> Sidebar Navigation </span><!--[--><!--]--><!--[--><div class="no-transition group" data-v-c40bc020><section class="VPSidebarItem level-0 has-active" data-v-c40bc020 data-v-b3fd67f8><div class="item" role="button" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><h2 class="text" data-v-b3fd67f8>User Guide</h2><!----></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../user-guide/getting-started.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Getting started</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../user-guide/understanding-findings.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Understanding findings</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../user-guide/run-comparison.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Run comparison mode</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../user-guide/mcp-tools.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>MCP tools reference</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../user-guide/alternative-log-retrieval.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Alternative log retrieval</p><!--]--></a><!----></div><!----></div><!--]--></div></section></div><!--]--><!--[--><!--]--></nav></aside><div class="VPContent has-sidebar" id="VPContent" data-v-5d98c3a5 data-v-1428d186><div class="VPDoc has-sidebar has-aside" data-v-1428d186 data-v-39a288b8><!--[--><!--]--><div class="container" data-v-39a288b8><div class="aside" data-v-39a288b8><div class="aside-curtain" data-v-39a288b8></div><div class="aside-container" data-v-39a288b8><div class="aside-content" data-v-39a288b8><div class="VPDocAside" data-v-39a288b8 data-v-3f215769><!--[--><!--]--><!--[--><!--]--><nav aria-labelledby="doc-outline-aria-label" class="VPDocAsideOutline" data-v-3f215769 data-v-a5bbad30><div class="content" data-v-a5bbad30><div class="outline-marker" data-v-a5bbad30></div><div aria-level="2" class="outline-title" id="doc-outline-aria-label" role="heading" data-v-a5bbad30>On this page</div><ul class="VPDocOutlineItem root" data-v-a5bbad30 data-v-b933a997><!--[--><!--]--></ul></div></nav><!--[--><!--]--><div class="spacer" data-v-3f215769></div><!--[--><!--]--><!----><!--[--><!--]--><!--[--><!--]--></div></div></div></div><div class="content" data-v-39a288b8><div class="content-container" data-v-39a288b8><!--[--><!--]--><main class="main" data-v-39a288b8><div style="position:relative;" class="vp-doc _docs_user-guide_alternative-log-retrieval" data-v-39a288b8><div><h1 id="alternative-ways-to-get-the-logs" tabindex="-1">Alternative ways to get the logs <a class="header-anchor" href="#alternative-ways-to-get-the-logs" aria-label="Permalink to "Alternative ways to get the logs""></a></h1><p><code>--shs-base-url</code> is the direct route: point SparkForensics at the History Server and it fetches the event log itself. This page covers what to do when that direct route isn't available.</p><h2 id="when-you-need-this" tabindex="-1">When you need this <a class="header-anchor" href="#when-you-need-this" aria-label="Permalink to "When you need this""></a></h2><p>The most common case is a Spark History Server (SHS) reachable only through an SSH bastion or jump host (the recipe below is the same regardless of what's issuing the SSH session), with the event logs themselves living on Kerberized HDFS and no direct HTTP path from your machine to SHS.</p><h2 id="why-shs-base-url-won-t-work-here" tabindex="-1">Why <code>--shs-base-url</code> won't work here <a class="header-anchor" href="#why-shs-base-url-won-t-work-here" aria-label="Permalink to "Why `--shs-base-url` won't work here""></a></h2><p>The CLI's <code>--shs-base-url</code>/<code>--app-id</code> flags and the MCP tools' <code>{ shsBaseUrl, appId }</code> source both make a direct HTTP request from wherever SparkForensics runs to the History Server's REST API. If that address is only reachable from inside a bastion, that request never lands: there's no <code>--via-ssh</code> flag or other way to route it through an SSH session.</p><h2 id="the-recipe" tabindex="-1">The recipe <a class="header-anchor" href="#the-recipe" aria-label="Permalink to "The recipe""></a></h2><ol><li><p>SSH into the edge node (through the bastion, or whatever gets you there).</p></li><li><p>Pull the event log off HDFS onto local disk on the edge node:</p><div class="language-bash vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">bash</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">hdfs</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> dfs</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;"> -get</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> /path/to/spark-events/application_XXXX_XXXX</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> /tmp/application_XXXX_XXXX</span></span></code></pre></div></li><li><p>Copy that file back to your own machine through the same bastion, e.g. with <code>scp</code> or <code>sftp</code>:</p><div class="language-bash vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">bash</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">scp</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> edge-node:/tmp/application_XXXX_XXXX</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> ./application_XXXX_XXXX</span></span></code></pre></div></li><li><p>Point SparkForensics at the local copy instead of a <code>shsBaseUrl</code>:</p><ul><li>CLI: <code>sparkforensics-analyze ./application_XXXX_XXXX</code></li><li>MCP: call a tool with <code>{ "source": { "path": "./application_XXXX_XXXX" } }</code></li></ul></li></ol><p>The file is the app's native input format either way: a single event-log file, optionally <code>.gz</code>/<code>.zstd</code>/<code>.lz4</code>/<code>.snappy</code>-compressed, or a rolling <code>eventlog_v2_*</code> directory pulled the same way. No conversion step needed.</p></div></div></main><footer class="VPDocFooter" data-v-39a288b8 data-v-e257564d><!--[--><!--]--><!----><nav class="prev-next" aria-labelledby="doc-footer-aria-label" data-v-e257564d><span class="visually-hidden" id="doc-footer-aria-label" data-v-e257564d>Pager</span><div class="pager" data-v-e257564d><a class="VPLink link pager-link prev" href="../user-guide/mcp-tools.html" data-v-e257564d><!--[--><span class="desc" data-v-e257564d>Previous page</span><span class="title" data-v-e257564d>MCP tools reference</span><!--]--></a></div><div class="pager" data-v-e257564d><!----></div></nav></footer><!--[--><!--]--></div></div></div><!--[--><!--]--></div></div><!----><!--[--><!--]--></div></div>
|
|
21
|
+
<div id="app"><div class="Layout" data-v-5d98c3a5><!--[--><!--]--><!--[--><span tabindex="-1" data-v-0b0ada53></span><a href="#VPContent" class="VPSkipLink visually-hidden" data-v-0b0ada53>Skip to content</a><!--]--><!----><header class="VPNav" data-v-5d98c3a5 data-v-ae24b3ad><div class="VPNavBar" data-v-ae24b3ad data-v-6aa21345><div class="wrapper" data-v-6aa21345><div class="container" data-v-6aa21345><div class="title" data-v-6aa21345><div class="VPNavBarTitle has-sidebar" data-v-6aa21345 data-v-1168a8e4><a class="title" href="../index.html" data-v-1168a8e4><!--[--><!--]--><!--[--><img class="VPImage logo" src="../favicon.svg" alt data-v-8426fc1a><!--]--><span data-v-1168a8e4>SparkForensics</span><!--[--><!--]--></a></div></div><div class="content" data-v-6aa21345><div class="content-body" data-v-6aa21345><!--[--><!--]--><div class="VPNavBarSearch search" data-v-6aa21345><!--[--><!----><div id="local-search"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><span class="vp-icon DocSearch-Search-Icon"></span><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"><kbd class="DocSearch-Button-Key"></kbd><kbd class="DocSearch-Button-Key">K</kbd></span></button></div><!--]--></div><nav aria-labelledby="main-nav-aria-label" class="VPNavBarMenu menu" data-v-6aa21345 data-v-dc692963><span id="main-nav-aria-label" class="visually-hidden" data-v-dc692963> Main Navigation </span><!--[--><!--[--><a class="VPLink link VPNavBarMenuLink" href="../user-guide/getting-started.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>User Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="../contributor-guide/development-setup.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Contributor Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="../tuning-reference/index.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Tuning Reference</span><!--]--></a><!--]--><!--]--></nav><!----><div class="VPNavBarAppearance appearance" data-v-6aa21345 data-v-6c893767><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-6c893767 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div><!----><div class="VPFlyout VPNavBarExtra extra" data-v-6aa21345 data-v-bb2aa2f0 data-v-cf11d7a2><button type="button" class="button" aria-haspopup="true" aria-expanded="false" aria-label="extra navigation" data-v-cf11d7a2><span class="vpi-more-horizontal icon" data-v-cf11d7a2></span></button><div class="menu" data-v-cf11d7a2><div class="VPMenu" data-v-cf11d7a2 data-v-b98bc113><!----><!--[--><!--[--><!----><div class="group" data-v-bb2aa2f0><div class="item appearance" data-v-bb2aa2f0><p class="label" data-v-bb2aa2f0>Appearance</p><div class="appearance-action" data-v-bb2aa2f0><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-bb2aa2f0 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div></div></div><!----><!--]--><!--]--></div></div></div><!--[--><!--]--><button type="button" class="VPNavBarHamburger hamburger" aria-label="mobile navigation" aria-expanded="false" aria-controls="VPNavScreen" data-v-6aa21345 data-v-e5dd9c1c><span class="container" data-v-e5dd9c1c><span class="top" data-v-e5dd9c1c></span><span class="middle" data-v-e5dd9c1c></span><span class="bottom" data-v-e5dd9c1c></span></span></button></div></div></div></div><div class="divider" data-v-6aa21345><div class="divider-line" data-v-6aa21345></div></div></div><!----></header><div class="VPLocalNav has-sidebar empty" data-v-5d98c3a5 data-v-a6f0e41e><div class="container" data-v-a6f0e41e><button class="menu" aria-expanded="false" aria-controls="VPSidebarNav" data-v-a6f0e41e><span class="vpi-align-left menu-icon" data-v-a6f0e41e></span><span class="menu-text" data-v-a6f0e41e>Menu</span></button><div class="VPLocalNavOutlineDropdown" style="--vp-vh:0px;" data-v-a6f0e41e data-v-8a42e2b4><button data-v-8a42e2b4>Return to top</button><!----></div></div></div><aside class="VPSidebar" data-v-5d98c3a5 data-v-319d5ca6><div class="curtain" data-v-319d5ca6></div><nav class="nav" id="VPSidebarNav" aria-labelledby="sidebar-aria-label" tabindex="-1" data-v-319d5ca6><span class="visually-hidden" id="sidebar-aria-label" data-v-319d5ca6> Sidebar Navigation </span><!--[--><!--]--><!--[--><div class="no-transition group" data-v-c40bc020><section class="VPSidebarItem level-0 has-active" data-v-c40bc020 data-v-b3fd67f8><div class="item" role="button" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><h2 class="text" data-v-b3fd67f8>User Guide</h2><!----></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../user-guide/getting-started.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Getting started</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../user-guide/understanding-findings.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Understanding findings</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../user-guide/run-comparison.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Run comparison mode</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../user-guide/mcp-tools.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>MCP tools reference</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../user-guide/alternative-log-retrieval.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Alternative log retrieval</p><!--]--></a><!----></div><!----></div><!--]--></div></section></div><!--]--><!--[--><!--]--></nav></aside><div class="VPContent has-sidebar" id="VPContent" data-v-5d98c3a5 data-v-1428d186><div class="VPDoc has-sidebar has-aside" data-v-1428d186 data-v-39a288b8><!--[--><!--]--><div class="container" data-v-39a288b8><div class="aside" data-v-39a288b8><div class="aside-curtain" data-v-39a288b8></div><div class="aside-container" data-v-39a288b8><div class="aside-content" data-v-39a288b8><div class="VPDocAside" data-v-39a288b8 data-v-3f215769><!--[--><!--]--><!--[--><!--]--><nav aria-labelledby="doc-outline-aria-label" class="VPDocAsideOutline" data-v-3f215769 data-v-a5bbad30><div class="content" data-v-a5bbad30><div class="outline-marker" data-v-a5bbad30></div><div aria-level="2" class="outline-title" id="doc-outline-aria-label" role="heading" data-v-a5bbad30>On this page</div><ul class="VPDocOutlineItem root" data-v-a5bbad30 data-v-b933a997><!--[--><!--]--></ul></div></nav><!--[--><!--]--><div class="spacer" data-v-3f215769></div><!--[--><!--]--><!----><!--[--><!--]--><!--[--><!--]--></div></div></div></div><div class="content" data-v-39a288b8><div class="content-container" data-v-39a288b8><!--[--><!--]--><main class="main" data-v-39a288b8><div style="position:relative;" class="vp-doc _docs_user-guide_alternative-log-retrieval" data-v-39a288b8><div><h1 id="alternative-ways-to-get-the-logs" tabindex="-1">Alternative ways to get the logs <a class="header-anchor" href="#alternative-ways-to-get-the-logs" aria-label="Permalink to "Alternative ways to get the logs""></a></h1><p><code>--shs-base-url</code> is the direct route: point SparkForensics at the History Server and it fetches the event log itself. This page covers what to do when that direct route isn't available.</p><h2 id="when-you-need-this" tabindex="-1">When you need this <a class="header-anchor" href="#when-you-need-this" aria-label="Permalink to "When you need this""></a></h2><p>The most common case is a Spark History Server (SHS) reachable only through an SSH bastion or jump host (the recipe below is the same regardless of what's issuing the SSH session), with the event logs themselves living on Kerberized HDFS and no direct HTTP path from your machine to SHS.</p><h2 id="why-shs-base-url-won-t-work-here" tabindex="-1">Why <code>--shs-base-url</code> won't work here <a class="header-anchor" href="#why-shs-base-url-won-t-work-here" aria-label="Permalink to "Why `--shs-base-url` won't work here""></a></h2><p>The CLI's <code>--shs-base-url</code>/<code>--app-id</code> flags and the MCP tools' <code>{ shsBaseUrl, appId }</code> source both make a direct HTTP request from wherever SparkForensics runs to the History Server's REST API. If that address is only reachable from inside a bastion, that request never lands: there's no <code>--via-ssh</code> flag or other way to route it through an SSH session.</p><h2 id="the-recipe" tabindex="-1">The recipe <a class="header-anchor" href="#the-recipe" aria-label="Permalink to "The recipe""></a></h2><ol><li><p>SSH into the edge node (through the bastion, or whatever gets you there).</p></li><li><p>Pull the event log off HDFS onto local disk on the edge node:</p><div class="language-bash vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">bash</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">hdfs</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> dfs</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;"> -get</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> /path/to/spark-events/application_XXXX_XXXX</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> /tmp/application_XXXX_XXXX</span></span></code></pre></div></li><li><p>Copy that file back to your own machine through the same bastion, e.g. with <code>scp</code> or <code>sftp</code>:</p><div class="language-bash vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">bash</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">scp</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> edge-node:/tmp/application_XXXX_XXXX</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> ./application_XXXX_XXXX</span></span></code></pre></div></li><li><p>Point SparkForensics at the local copy instead of a <code>shsBaseUrl</code>:</p><ul><li>Browser: drop the file onto the landing page, same as any other run (see <a href="./getting-started.html">Getting started</a>).</li><li>CLI: <code>npx sparkforensics-analyze ./application_XXXX_XXXX</code> (full flag list in <a href="./getting-started.html#ci-and-automation">Getting started</a>).</li><li>MCP: call a tool with <code>{ "source": { "path": "./application_XXXX_XXXX" } }</code> (tool reference in <a href="./mcp-tools.html">MCP tools reference</a>).</li></ul></li></ol><p>The file is the app's native input format either way: a single event-log file, optionally <code>.gz</code>/<code>.zstd</code>/<code>.lz4</code>/<code>.snappy</code>-compressed, or a rolling <code>eventlog_v2_*</code> directory pulled the same way. No conversion step needed.</p></div></div></main><footer class="VPDocFooter" data-v-39a288b8 data-v-e257564d><!--[--><!--]--><!----><nav class="prev-next" aria-labelledby="doc-footer-aria-label" data-v-e257564d><span class="visually-hidden" id="doc-footer-aria-label" data-v-e257564d>Pager</span><div class="pager" data-v-e257564d><a class="VPLink link pager-link prev" href="../user-guide/mcp-tools.html" data-v-e257564d><!--[--><span class="desc" data-v-e257564d>Previous page</span><span class="title" data-v-e257564d>MCP tools reference</span><!--]--></a></div><div class="pager" data-v-e257564d><!----></div></nav></footer><!--[--><!--]--></div></div></div><!--[--><!--]--></div></div><!----><!--[--><!--]--></div></div>
|
|
22
22
|
|
|
23
23
|
|
|
24
24
|
</body>
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
<title>Getting started | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="../assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="../vp-icons.css" as="style">
|
|
11
11
|
|
|
12
12
|
|
|
@@ -18,9 +18,9 @@
|
|
|
18
18
|
<script id="check-mac-os">document.documentElement.classList.toggle("mac",/Mac|iPhone|iPod|iPad/i.test(navigator.platform));</script>
|
|
19
19
|
</head>
|
|
20
20
|
<body>
|
|
21
|
-
<div id="app"><div class="Layout" data-v-5d98c3a5><!--[--><!--]--><!--[--><span tabindex="-1" data-v-0b0ada53></span><a href="#VPContent" class="VPSkipLink visually-hidden" data-v-0b0ada53>Skip to content</a><!--]--><!----><header class="VPNav" data-v-5d98c3a5 data-v-ae24b3ad><div class="VPNavBar" data-v-ae24b3ad data-v-6aa21345><div class="wrapper" data-v-6aa21345><div class="container" data-v-6aa21345><div class="title" data-v-6aa21345><div class="VPNavBarTitle has-sidebar" data-v-6aa21345 data-v-1168a8e4><a class="title" href="../index.html" data-v-1168a8e4><!--[--><!--]--><!--[--><img class="VPImage logo" src="../favicon.svg" alt data-v-8426fc1a><!--]--><span data-v-1168a8e4>SparkForensics</span><!--[--><!--]--></a></div></div><div class="content" data-v-6aa21345><div class="content-body" data-v-6aa21345><!--[--><!--]--><div class="VPNavBarSearch search" data-v-6aa21345><!--[--><!----><div id="local-search"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><span class="vp-icon DocSearch-Search-Icon"></span><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"><kbd class="DocSearch-Button-Key"></kbd><kbd class="DocSearch-Button-Key">K</kbd></span></button></div><!--]--></div><nav aria-labelledby="main-nav-aria-label" class="VPNavBarMenu menu" data-v-6aa21345 data-v-dc692963><span id="main-nav-aria-label" class="visually-hidden" data-v-dc692963> Main Navigation </span><!--[--><!--[--><a class="VPLink link VPNavBarMenuLink active" href="../user-guide/getting-started.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>User Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="../contributor-guide/development-setup.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Contributor Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="../tuning-reference/index.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Tuning Reference</span><!--]--></a><!--]--><!--]--></nav><!----><div class="VPNavBarAppearance appearance" data-v-6aa21345 data-v-6c893767><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-6c893767 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div><!----><div class="VPFlyout VPNavBarExtra extra" data-v-6aa21345 data-v-bb2aa2f0 data-v-cf11d7a2><button type="button" class="button" aria-haspopup="true" aria-expanded="false" aria-label="extra navigation" data-v-cf11d7a2><span class="vpi-more-horizontal icon" data-v-cf11d7a2></span></button><div class="menu" data-v-cf11d7a2><div class="VPMenu" data-v-cf11d7a2 data-v-b98bc113><!----><!--[--><!--[--><!----><div class="group" data-v-bb2aa2f0><div class="item appearance" data-v-bb2aa2f0><p class="label" data-v-bb2aa2f0>Appearance</p><div class="appearance-action" data-v-bb2aa2f0><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-bb2aa2f0 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div></div></div><!----><!--]--><!--]--></div></div></div><!--[--><!--]--><button type="button" class="VPNavBarHamburger hamburger" aria-label="mobile navigation" aria-expanded="false" aria-controls="VPNavScreen" data-v-6aa21345 data-v-e5dd9c1c><span class="container" data-v-e5dd9c1c><span class="top" data-v-e5dd9c1c></span><span class="middle" data-v-e5dd9c1c></span><span class="bottom" data-v-e5dd9c1c></span></span></button></div></div></div></div><div class="divider" data-v-6aa21345><div class="divider-line" data-v-6aa21345></div></div></div><!----></header><div class="VPLocalNav has-sidebar empty" data-v-5d98c3a5 data-v-a6f0e41e><div class="container" data-v-a6f0e41e><button class="menu" aria-expanded="false" aria-controls="VPSidebarNav" data-v-a6f0e41e><span class="vpi-align-left menu-icon" data-v-a6f0e41e></span><span class="menu-text" data-v-a6f0e41e>Menu</span></button><div class="VPLocalNavOutlineDropdown" style="--vp-vh:0px;" data-v-a6f0e41e data-v-8a42e2b4><button data-v-8a42e2b4>Return to top</button><!----></div></div></div><aside class="VPSidebar" data-v-5d98c3a5 data-v-319d5ca6><div class="curtain" data-v-319d5ca6></div><nav class="nav" id="VPSidebarNav" aria-labelledby="sidebar-aria-label" tabindex="-1" data-v-319d5ca6><span class="visually-hidden" id="sidebar-aria-label" data-v-319d5ca6> Sidebar Navigation </span><!--[--><!--]--><!--[--><div class="no-transition group" data-v-c40bc020><section class="VPSidebarItem level-0 has-active" data-v-c40bc020 data-v-b3fd67f8><div class="item" role="button" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><h2 class="text" data-v-b3fd67f8>User Guide</h2><!----></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../user-guide/getting-started.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Getting started</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../user-guide/understanding-findings.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Understanding findings</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../user-guide/run-comparison.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Run comparison mode</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../user-guide/mcp-tools.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>MCP tools reference</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../user-guide/alternative-log-retrieval.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Alternative log retrieval</p><!--]--></a><!----></div><!----></div><!--]--></div></section></div><!--]--><!--[--><!--]--></nav></aside><div class="VPContent has-sidebar" id="VPContent" data-v-5d98c3a5 data-v-1428d186><div class="VPDoc has-sidebar has-aside" data-v-1428d186 data-v-39a288b8><!--[--><!--]--><div class="container" data-v-39a288b8><div class="aside" data-v-39a288b8><div class="aside-curtain" data-v-39a288b8></div><div class="aside-container" data-v-39a288b8><div class="aside-content" data-v-39a288b8><div class="VPDocAside" data-v-39a288b8 data-v-3f215769><!--[--><!--]--><!--[--><!--]--><nav aria-labelledby="doc-outline-aria-label" class="VPDocAsideOutline" data-v-3f215769 data-v-a5bbad30><div class="content" data-v-a5bbad30><div class="outline-marker" data-v-a5bbad30></div><div aria-level="2" class="outline-title" id="doc-outline-aria-label" role="heading" data-v-a5bbad30>On this page</div><ul class="VPDocOutlineItem root" data-v-a5bbad30 data-v-b933a997><!--[--><!--]--></ul></div></nav><!--[--><!--]--><div class="spacer" data-v-3f215769></div><!--[--><!--]--><!----><!--[--><!--]--><!--[--><!--]--></div></div></div></div><div class="content" data-v-39a288b8><div class="content-container" data-v-39a288b8><!--[--><!--]--><main class="main" data-v-39a288b8><div style="position:relative;" class="vp-doc _docs_user-guide_getting-started" data-v-39a288b8><div><h1 id="getting-started" tabindex="-1">Getting started <a class="header-anchor" href="#getting-started" aria-label="Permalink to "Getting started""></a></h1><p>SparkForensics reads a Spark History Server event log and turns it into a dashboard of flagged bottlenecks. No server or account required.</p><h2 id="load-a-run" tabindex="-1">Load a run <a class="header-anchor" href="#load-a-run" aria-label="Permalink to "Load a run""></a></h2><p>Drop a log file onto the landing page, or click <strong>Choose file</strong> to pick one. Parsing runs in a background worker, off the browser's main thread, so a multi-hundred-megabyte event stream doesn't freeze the tab.</p><p>The app takes a newline-delimited JSON event log (one JSON event per line, the format Spark writes to <code>spark.eventLog.dir</code>), either plain or gzip/Zstandard/LZ4/Snappy-compressed.</p><p>The same landing page has two other ways in: <strong>Choose rolling-log folder</strong> for an <code>eventlog_v2_*</code> rolling directory, and <strong>Fetch from Spark History Server</strong> to pull an application from a reachable History Server (local-server mode only).</p><p>Can't reach the History Server directly (it's only reachable through an SSH bastion)? See <a href="./alternative-log-retrieval.html">Alternative ways to get the logs</a>.</p><p>A History Server export needs one manual step. <code>GET /api/v1/applications/<appId>/logs</code> hands back a zip, and the drop zone does not unwrap zip containers. Extract the <code>.zstd</code> event log from that zip yourself and drop the extracted file in. It needs no further decompressing.</p><p>Files you have already loaded stay listed under <strong>Recent files</strong> on the landing page.</p><h2 id="reading-the-dashboard" tabindex="-1">Reading the dashboard <a class="header-anchor" href="#reading-the-dashboard" aria-label="Permalink to "Reading the dashboard""></a></h2><p>When parsing finishes, the dashboard shows a board of widgets, each in its own card with a title. Every widget that flags a problem uses the same convention: a colored impact dot (critical / warning / info) and an ALL-CAPS tag for the bottleneck category (<code>SKEW</code>, <code>SPILL</code>, <code>GC</code>, and so on: see <a href="./understanding-findings.html">Understanding findings</a>). Widgets list every affected stage, not just the worst one. The dot's color tracks how much run time the finding could save you rather than how unusual the metric looks, so a small-looking anomaly with a big payoff can outrank a dramatic one that would barely move your run time.</p><p>A run scorecard (wall-clock, efficiency, wastage) always shows at the top. Below it, two tabs split the rest of the board:</p><ol><li><strong>Findings</strong>: every flagged finding and its detail widget, grouped by impact band (Critical, Warning, Info). Within a band, a recommendation row for a bottleneck type collapses into a summary row when it fires more than once; clicking the summary row expands its full list, and clicking any single row or widget jumps straight to that finding. Below the impact-band groups, memory and core-usage utilization always show, even on a clean run, and widgets that found nothing fold away into a "Clean checks" disclosure. This is the tab you land on.</li><li><strong>Full app report</strong>: the wall-clock and executor timelines, the stage table, and the reference-only cards.</li></ol><h3 id="advanced-view" tabindex="-1">Advanced view <a class="header-anchor" href="#advanced-view" aria-label="Permalink to "Advanced view""></a></h3><p>The topbar has an <strong>Advanced view</strong> toggle. It's off by default, which keeps each widget to the finding itself and what to do about it. Turn it on to also show confidence levels, supporting evidence, and documentation links for each finding, plus a few extra table columns. Your choice is remembered across runs.</p><h2 id="compare-two-runs" tabindex="-1">Compare two runs <a class="header-anchor" href="#compare-two-runs" aria-label="Permalink to "Compare two runs""></a></h2><p>To compare a baseline run against a candidate, say to check whether a tuning change helped, see <a href="./run-comparison.html">Run comparison mode</a>.</p><h2 id="ci-and-automation" tabindex="-1">CI and automation <a class="header-anchor" href="#ci-and-automation" aria-label="Permalink to "CI and automation""></a></h2><p>In a pipeline or a script, use the <code>analyze</code> CLI instead of the browser dashboard. It parses the same event logs and exits non-zero when a run crosses a threshold you set, so it can gate a build:</p><div class="language-sh vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">sh</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">npx</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> sparkforensics-analyze</span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;"> <</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">file</span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">|</span><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">dir</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">> [--max-runtime </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">ms]</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> [--max-skew </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">ratio]</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;"> \</span></span>
|
|
21
|
+
<div id="app"><div class="Layout" data-v-5d98c3a5><!--[--><!--]--><!--[--><span tabindex="-1" data-v-0b0ada53></span><a href="#VPContent" class="VPSkipLink visually-hidden" data-v-0b0ada53>Skip to content</a><!--]--><!----><header class="VPNav" data-v-5d98c3a5 data-v-ae24b3ad><div class="VPNavBar" data-v-ae24b3ad data-v-6aa21345><div class="wrapper" data-v-6aa21345><div class="container" data-v-6aa21345><div class="title" data-v-6aa21345><div class="VPNavBarTitle has-sidebar" data-v-6aa21345 data-v-1168a8e4><a class="title" href="../index.html" data-v-1168a8e4><!--[--><!--]--><!--[--><img class="VPImage logo" src="../favicon.svg" alt data-v-8426fc1a><!--]--><span data-v-1168a8e4>SparkForensics</span><!--[--><!--]--></a></div></div><div class="content" data-v-6aa21345><div class="content-body" data-v-6aa21345><!--[--><!--]--><div class="VPNavBarSearch search" data-v-6aa21345><!--[--><!----><div id="local-search"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><span class="vp-icon DocSearch-Search-Icon"></span><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"><kbd class="DocSearch-Button-Key"></kbd><kbd class="DocSearch-Button-Key">K</kbd></span></button></div><!--]--></div><nav aria-labelledby="main-nav-aria-label" class="VPNavBarMenu menu" data-v-6aa21345 data-v-dc692963><span id="main-nav-aria-label" class="visually-hidden" data-v-dc692963> Main Navigation </span><!--[--><!--[--><a class="VPLink link VPNavBarMenuLink active" href="../user-guide/getting-started.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>User Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="../contributor-guide/development-setup.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Contributor Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="../tuning-reference/index.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Tuning Reference</span><!--]--></a><!--]--><!--]--></nav><!----><div class="VPNavBarAppearance appearance" data-v-6aa21345 data-v-6c893767><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-6c893767 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div><!----><div class="VPFlyout VPNavBarExtra extra" data-v-6aa21345 data-v-bb2aa2f0 data-v-cf11d7a2><button type="button" class="button" aria-haspopup="true" aria-expanded="false" aria-label="extra navigation" data-v-cf11d7a2><span class="vpi-more-horizontal icon" data-v-cf11d7a2></span></button><div class="menu" data-v-cf11d7a2><div class="VPMenu" data-v-cf11d7a2 data-v-b98bc113><!----><!--[--><!--[--><!----><div class="group" data-v-bb2aa2f0><div class="item appearance" data-v-bb2aa2f0><p class="label" data-v-bb2aa2f0>Appearance</p><div class="appearance-action" data-v-bb2aa2f0><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-bb2aa2f0 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div></div></div><!----><!--]--><!--]--></div></div></div><!--[--><!--]--><button type="button" class="VPNavBarHamburger hamburger" aria-label="mobile navigation" aria-expanded="false" aria-controls="VPNavScreen" data-v-6aa21345 data-v-e5dd9c1c><span class="container" data-v-e5dd9c1c><span class="top" data-v-e5dd9c1c></span><span class="middle" data-v-e5dd9c1c></span><span class="bottom" data-v-e5dd9c1c></span></span></button></div></div></div></div><div class="divider" data-v-6aa21345><div class="divider-line" data-v-6aa21345></div></div></div><!----></header><div class="VPLocalNav has-sidebar empty" data-v-5d98c3a5 data-v-a6f0e41e><div class="container" data-v-a6f0e41e><button class="menu" aria-expanded="false" aria-controls="VPSidebarNav" data-v-a6f0e41e><span class="vpi-align-left menu-icon" data-v-a6f0e41e></span><span class="menu-text" data-v-a6f0e41e>Menu</span></button><div class="VPLocalNavOutlineDropdown" style="--vp-vh:0px;" data-v-a6f0e41e data-v-8a42e2b4><button data-v-8a42e2b4>Return to top</button><!----></div></div></div><aside class="VPSidebar" data-v-5d98c3a5 data-v-319d5ca6><div class="curtain" data-v-319d5ca6></div><nav class="nav" id="VPSidebarNav" aria-labelledby="sidebar-aria-label" tabindex="-1" data-v-319d5ca6><span class="visually-hidden" id="sidebar-aria-label" data-v-319d5ca6> Sidebar Navigation </span><!--[--><!--]--><!--[--><div class="no-transition group" data-v-c40bc020><section class="VPSidebarItem level-0 has-active" data-v-c40bc020 data-v-b3fd67f8><div class="item" role="button" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><h2 class="text" data-v-b3fd67f8>User Guide</h2><!----></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../user-guide/getting-started.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Getting started</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../user-guide/understanding-findings.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Understanding findings</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../user-guide/run-comparison.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Run comparison mode</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../user-guide/mcp-tools.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>MCP tools reference</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../user-guide/alternative-log-retrieval.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Alternative log retrieval</p><!--]--></a><!----></div><!----></div><!--]--></div></section></div><!--]--><!--[--><!--]--></nav></aside><div class="VPContent has-sidebar" id="VPContent" data-v-5d98c3a5 data-v-1428d186><div class="VPDoc has-sidebar has-aside" data-v-1428d186 data-v-39a288b8><!--[--><!--]--><div class="container" data-v-39a288b8><div class="aside" data-v-39a288b8><div class="aside-curtain" data-v-39a288b8></div><div class="aside-container" data-v-39a288b8><div class="aside-content" data-v-39a288b8><div class="VPDocAside" data-v-39a288b8 data-v-3f215769><!--[--><!--]--><!--[--><!--]--><nav aria-labelledby="doc-outline-aria-label" class="VPDocAsideOutline" data-v-3f215769 data-v-a5bbad30><div class="content" data-v-a5bbad30><div class="outline-marker" data-v-a5bbad30></div><div aria-level="2" class="outline-title" id="doc-outline-aria-label" role="heading" data-v-a5bbad30>On this page</div><ul class="VPDocOutlineItem root" data-v-a5bbad30 data-v-b933a997><!--[--><!--]--></ul></div></nav><!--[--><!--]--><div class="spacer" data-v-3f215769></div><!--[--><!--]--><!----><!--[--><!--]--><!--[--><!--]--></div></div></div></div><div class="content" data-v-39a288b8><div class="content-container" data-v-39a288b8><!--[--><!--]--><main class="main" data-v-39a288b8><div style="position:relative;" class="vp-doc _docs_user-guide_getting-started" data-v-39a288b8><div><h1 id="getting-started" tabindex="-1">Getting started <a class="header-anchor" href="#getting-started" aria-label="Permalink to "Getting started""></a></h1><p>SparkForensics reads a Spark History Server event log and turns it into a dashboard of flagged bottlenecks. No install and no account: open the app in a browser and drop a log in.</p><h2 id="load-a-run" tabindex="-1">Load a run <a class="header-anchor" href="#load-a-run" aria-label="Permalink to "Load a run""></a></h2><p>Drop a log file onto the landing page, or click <strong>Choose file</strong> to pick one. Parsing runs in a background worker, off the browser's main thread, so a multi-hundred-megabyte event stream doesn't freeze the tab.</p><p>No log of your own yet? Click <strong>Try a sample run</strong> on the landing page to load a bundled example run and see a populated dashboard right away.</p><p>The app takes a newline-delimited JSON event log (one JSON event per line, the format Spark writes to <code>spark.eventLog.dir</code>), either plain or gzip/Zstandard/LZ4/Snappy-compressed.</p><p>Click <strong>Other sources</strong> on the landing page for two more ways in:</p><ul><li><strong>Choose rolling-log folder</strong>, for an <code>eventlog_v2_*</code> rolling directory. Drop the folder or point the picker at it; the app reassembles its parts in order before parsing.</li><li><strong>Fetch from Spark History Server</strong>, to pull an application straight from a reachable History Server. This needs <strong>local-server mode</strong> (see below) and an application ID in one of the forms Spark itself uses: <code>application_<timestamp>_<id></code>, <code>local-<timestamp></code>, <code>app-<id></code>, <code>spark-<id></code>, or <code>driver-<id></code>. If the fetch fails, the panel names the problem (server unreachable, application not found, local server not running, and so on) and suggests what to try next.</li></ul><p>Can't reach the History Server directly (it's only reachable through an SSH bastion)? See <a href="./alternative-log-retrieval.html">Alternative ways to get the logs</a>.</p><p>A History Server export needs one manual step. <code>GET /api/v1/applications/<appId>/logs</code> hands back a zip, and the drop zone does not unwrap zip containers. Extract the <code>.zstd</code> event log from that zip yourself and drop the extracted file in. It needs no further decompressing.</p><p>Files you have already loaded stay listed under <strong>Recent files</strong> on the landing page.</p><h3 id="local-server-mode" tabindex="-1">Local-server mode <a class="header-anchor" href="#local-server-mode" aria-label="Permalink to "Local-server mode""></a></h3><p>The plain browser app can't fetch from a History Server itself: the browser's CORS policy blocks a cross-origin request like that, and there's no server on the other end to proxy it. Local-server mode adds that server. Run it with:</p><div class="language-sh vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">sh</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">npx</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> sparkforensics-server</span></span></code></pre></div><p>It listens on <code>http://127.0.0.1:4173</code> by default and binds to localhost only. Open that URL instead of the static build, and the <strong>Fetch from Spark History Server</strong> panel works because requests now go server-to-server. Full setup and its port/environment-variable overrides are in the <a href="https://github.com/shuffle-works/sparkforensics#readme" target="_blank" rel="noreferrer">project README</a>, which also covers what changes for a static (no-server) deploy.</p><h2 id="reading-the-dashboard" tabindex="-1">Reading the dashboard <a class="header-anchor" href="#reading-the-dashboard" aria-label="Permalink to "Reading the dashboard""></a></h2><p>When parsing finishes, the dashboard shows a board of widgets, each in its own card with a title. Every widget that flags a problem uses the same convention: a colored impact dot (critical / warning / info) and an ALL-CAPS tag for the bottleneck category (<code>SKEW</code>, <code>SPILL</code>, <code>GC</code>, and so on: see <a href="./understanding-findings.html">Understanding findings</a>). Widgets list every affected stage, not just the worst one. The dot's color tracks how much run time the finding could save you rather than how unusual the metric looks, so a small-looking anomaly with a big payoff can outrank a dramatic one that would barely move your run time.</p><p>A run scorecard (wall-clock, efficiency, wastage) always shows at the top. Below it, two tabs split the rest of the board:</p><ol><li><strong>Findings</strong>: every flagged finding and its detail widget, grouped by impact band (Critical, Warning, Info). Within a band, a recommendation row for a bottleneck type collapses into a summary row when it fires more than once; clicking the summary row expands its full list, and clicking any single row or widget jumps straight to that finding. Below the impact-band groups, memory and core-usage utilization always show, even on a clean run. Widgets that found nothing fold away into a "Clean checks" disclosure. This is the tab you land on.</li><li><strong>Full app report</strong>: the wall-clock and executor timelines, the stage table, and the reference-only cards.</li></ol><p>Click a finding's documentation link (or the topbar's <strong>Docs</strong> button) to open the reference material in a slide-in panel beside the dashboard: the dashboard stays visible and interactive, so you can check a metric against the reference without losing your place.</p><h3 id="advanced-view" tabindex="-1">Advanced view <a class="header-anchor" href="#advanced-view" aria-label="Permalink to "Advanced view""></a></h3><p>The topbar has an <strong>Advanced view</strong> toggle. It's off by default, which keeps each widget to the finding itself and what to do about it. Turn it on to also show confidence levels, supporting evidence, and documentation links for each finding, plus a few extra table columns. Your choice is remembered across runs.</p><h3 id="the-rest-of-the-topbar" tabindex="-1">The rest of the topbar <a class="header-anchor" href="#the-rest-of-the-topbar" aria-label="Permalink to "The rest of the topbar""></a></h3><p>Once a run is loaded, the topbar also carries a few more controls. <strong>New analysis</strong> goes back to the landing page to load another run. <strong>Plan graph</strong> opens an interactive node-and-edge view of the run's SQL execution plan, filterable down to I/O operators (scan, exchange), a broader "basic" set, or every operator. <strong>Export evidence</strong> downloads the current run's findings as a portable Markdown or JSON report, the same shape the CLI and MCP tools produce; turn on <strong>Redact identifiers</strong> first if the report is headed outside the environment that produced it, since that pseudonymizes the app id and any host/IP tokens. <strong>Keyboard shortcuts</strong> (press <code>?</code> from anywhere) lists every shortcut. Rounding it out: a <strong>Docs</strong> link and a theme toggle.</p><h2 id="compare-two-runs" tabindex="-1">Compare two runs <a class="header-anchor" href="#compare-two-runs" aria-label="Permalink to "Compare two runs""></a></h2><p>To compare a baseline run against a candidate, say to check whether a tuning change helped, see <a href="./run-comparison.html">Run comparison mode</a>.</p><h2 id="ci-and-automation" tabindex="-1">CI and automation <a class="header-anchor" href="#ci-and-automation" aria-label="Permalink to "CI and automation""></a></h2><p>In a pipeline or a script, use the <code>analyze</code> CLI instead of the browser dashboard. It parses the same event logs and exits non-zero when a run crosses a threshold you set, so it can gate a build:</p><div class="language-sh vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">sh</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">npx</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> sparkforensics-analyze</span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;"> <</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">file</span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">|</span><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">dir</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">> [--max-runtime </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">ms]</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> [--max-skew </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">ratio]</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;"> \</span></span>
|
|
22
22
|
<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> [--max-spill </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">gb]</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> [--max-failed-task-rate </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">pct]</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> [--min-efficiency </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">pct]</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;"> \</span></span>
|
|
23
|
-
<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> [--out </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">path]</span></span></code></pre></div><p>Point it at a single event log or a directory of them. Pass <code>--out</code> to also write the findings to a file.</p><p>Pass <code>--export-html <dir></code> to write a self-contained HTML dashboard for the run into <code><dir></code> (which must not already exist or must be empty). Open <code><dir>/index.html</code> directly in a browser over <code>file://</code>, with no server, to get the same interactive dashboard offline. This makes it easy to archive or share a run. <code>--redact</code> applies to the exported report too.</p><p>The CLI also supports fetching a run directly from a reachable Spark History Server (<code>--shs-base-url</code>/<code>--app-id</code>/<code>--attempt-id</code>) instead of a local file, comparing a candidate run against a baseline with regression gating (<code>--baseline</code>/<code>--max-regression-pct</code>/<code>--regression-metric</code>/ <code>--fail-on-introduced</code>), redacting the app id and any host/IP tokens before sharing output (<code>--redact</code>), and narrowing the findings to certain impact bands, types, or a stage (<code>--impact</code>/<code>--type</code>/<code>--stage</code>). Run it with <code>--help</code> for the full flag list.</p><p>Same caveat as above: if the History Server is only reachable through an SSH bastion, <code>--shs-base-url</code> can't reach it either: see <a href="./alternative-log-retrieval.html">Alternative ways to get the logs</a>.</p><p>Running in Airflow instead of a plain CI pipeline? See <a href="https://github.com/shuffle-works/sparkforensics-operator" target="_blank" rel="noreferrer">sparkforensics-operator</a>, an Airflow operator that wraps the CLI and acts on the result after each Spark job, so you don't have to wire up the call yourself.</p></div></div></main><footer class="VPDocFooter" data-v-39a288b8 data-v-e257564d><!--[--><!--]--><!----><nav class="prev-next" aria-labelledby="doc-footer-aria-label" data-v-e257564d><span class="visually-hidden" id="doc-footer-aria-label" data-v-e257564d>Pager</span><div class="pager" data-v-e257564d><!----></div><div class="pager" data-v-e257564d><a class="VPLink link pager-link next" href="../user-guide/understanding-findings.html" data-v-e257564d><!--[--><span class="desc" data-v-e257564d>Next page</span><span class="title" data-v-e257564d>Understanding findings</span><!--]--></a></div></nav></footer><!--[--><!--]--></div></div></div><!--[--><!--]--></div></div><!----><!--[--><!--]--></div></div>
|
|
23
|
+
<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> [--out </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">path]</span></span></code></pre></div><p>Point it at a single event log or a directory of them. Pass <code>--out</code> to also write the findings to a file.</p><p>Pass <code>--export-html <dir></code> to write a self-contained HTML dashboard for the run into <code><dir></code> (which must not already exist or must be empty). Open <code><dir>/index.html</code> directly in a browser over <code>file://</code>, with no server, to get the same interactive dashboard offline. This makes it easy to archive or share a run. <code>--redact</code> applies to the exported report too.</p><p>The CLI also supports fetching a run directly from a reachable Spark History Server (<code>--shs-base-url</code>/<code>--app-id</code>/<code>--attempt-id</code>) instead of a local file, comparing a candidate run against a baseline with regression gating (<code>--baseline</code>/<code>--max-regression-pct</code>/<code>--regression-metric</code>/ <code>--fail-on-introduced</code>), redacting the app id and any host/IP tokens before sharing output (<code>--redact</code>), and narrowing the findings to certain impact bands, types, or a stage (<code>--impact</code>/<code>--type</code>/<code>--stage</code>). Run it with <code>--help</code> for the full flag list.</p><p>Same caveat as above: if the History Server is only reachable through an SSH bastion, <code>--shs-base-url</code> can't reach it either: see <a href="./alternative-log-retrieval.html">Alternative ways to get the logs</a>.</p><p>Running in Airflow instead of a plain CI pipeline? See <a href="https://github.com/shuffle-works/sparkforensics-operator" target="_blank" rel="noreferrer">sparkforensics-operator</a>, an Airflow operator that wraps the CLI and acts on the result after each Spark job, so you don't have to wire up the call yourself.</p><p>Want an AI assistant to diagnose a run directly, without the dashboard or a CI gate? See <a href="./mcp-tools.html">MCP tools reference</a>.</p></div></div></main><footer class="VPDocFooter" data-v-39a288b8 data-v-e257564d><!--[--><!--]--><!----><nav class="prev-next" aria-labelledby="doc-footer-aria-label" data-v-e257564d><span class="visually-hidden" id="doc-footer-aria-label" data-v-e257564d>Pager</span><div class="pager" data-v-e257564d><!----></div><div class="pager" data-v-e257564d><a class="VPLink link pager-link next" href="../user-guide/understanding-findings.html" data-v-e257564d><!--[--><span class="desc" data-v-e257564d>Next page</span><span class="title" data-v-e257564d>Understanding findings</span><!--]--></a></div></nav></footer><!--[--><!--]--></div></div></div><!--[--><!--]--></div></div><!----><!--[--><!--]--></div></div>
|
|
24
24
|
|
|
25
25
|
|
|
26
26
|
</body>
|