sparklens 0.1.0__tar.gz

sparklens-0.1.0/PKG-INFO

@@ -0,0 +1,496 @@
+Metadata-Version: 2.4
+Name: sparklens
+Version: 0.1.0
+Summary: Spark Physical Plan Analyzer and Visualizer
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: altair
+Requires-Dist: matplotlib
+Requires-Dist: networkx
+Requires-Dist: pydot
+Requires-Dist: pyvis
+Requires-Dist: google-genai
+Requires-Dist: python-dotenv
+
+# SparkLens
+
+## Deterministic Spark Execution Plan Analyzer with AI-Assisted Explanations
+
+SparkLens is a tool for analyzing Apache Spark Physical Execution Plans.
+
+It transforms raw Spark execution plans into:
+
+* Structured execution DAGs
+* Deterministic performance findings
+* Visual execution graphs
+* Markdown and HTML reports
+* AI-generated explanations powered by Gemini
+
+SparkLens is designed around a simple principle:
+
+> AI should explain analysis, not replace analysis.
+
+Unlike tools that send raw execution plans directly to an LLM, SparkLens first performs deterministic rule-based analysis and then uses AI to explain the findings.
+
+---
+
+# Why SparkLens?
+
+Spark execution plans are powerful but difficult to interpret.
+
+Even moderately complex plans may contain:
+
+* Broadcast joins
+* Shuffle exchanges
+* Multi-stage aggregations
+* Sorting operations
+* Adaptive Query Execution (AQE)
+* Partitioning strategies
+* Stage boundaries
+
+Understanding:
+
+* Why a query is slow
+* Where data is shuffled
+* Which join strategy Spark selected
+* Where parallelism is lost
+* How execution stages are formed
+
+often requires significant Spark expertise.
+
+SparkLens automates this process and presents the results in a structured, explainable format.
+
+---
+
+# Core Features
+
+## Execution Plan Parsing
+
+Parse Spark Physical Execution Plans into structured operator representations.
+
+---
+
+## Execution DAG Construction
+
+Convert execution plans into a directed acyclic graph representing execution flow.
+
+---
+
+## Deterministic Rule Engine
+
+Detects Spark performance patterns and execution characteristics including:
+
+* Broadcast Hash Joins
+* Shuffle Operations
+* Stage Boundaries
+* Single Partition Bottlenecks
+* Aggregation Pipelines
+* Sorting Operations
+* AQE Usage
+* Partitioning Strategies
+* Performance Anti-Patterns
+
+---
+
+## AI-Assisted Explanations
+
+Generate:
+
+* Executive Summaries
+* Execution Walkthroughs
+* Performance Impact Analysis
+* Optimization Recommendations
+
+using Google Gemini.
+
+The AI layer receives structured findings generated by SparkLens and is instructed to explain those findings rather than independently analyze the plan.
+
+---
+
+## Visualization
+
+Generate execution DAG visualizations as PNG images.
+
+Visualization highlights operators associated with analysis findings.
+
+---
+
+## Reporting
+
+Generate:
+
+* Markdown Reports
+* HTML Reports
+
+containing both deterministic analysis and AI-generated explanations.
+
+---
+
+# Analysis Philosophy
+
+SparkLens follows a deterministic-first architecture.
+
+Instead of sending a raw execution plan directly to an LLM:
+
+```text
+Spark Plan
+    ↓
+LLM
+    ↓
+Explanation
+```
+
+SparkLens performs structured analysis first:
+
+```text
+Spark Plan
+    ↓
+Parser
+    ↓
+Execution DAG
+    ↓
+Rule Engine
+    ↓
+Analysis Report
+    ↓
+Recommendation Engine
+    ↓
+AI Explanation
+```
+
+Benefits:
+
+* Reduced hallucinations
+* Consistent findings
+* Explainable analysis
+* Reproducible results
+* Better separation between analysis and explanation
+
+---
+
+# Evaluation
+
+SparkLens findings are scored against a fixed ground-truth dataset. The evaluation runs fully offline (no API calls, no model downloads) and is deterministic: the same inputs always give the same numbers.
+
+## What is measured
+
+* `evaluation/dataset_expected.csv` — 50 real Spark physical plans (49 rendered with Spark 3.5 `explain("formatted")`, 1 Databricks Photon plan), each with the analysis a correct analyzer should produce, written as plain sentences (316 atomic claims)
+* `evaluation/dataset_predicted.csv` — what SparkLens reports for each plan (`sparklens analyze <plan> --eval`: rule engine only, AI disabled)
+
+Both paragraphs are split into atomic claims and compared claim by claim:
+
+* **Precision** — share of SparkLens claims backed by the ground truth (false alarms lower it)
+* **Recall** — share of ground-truth claims that SparkLens states (misses lower it)
+* **F1** — balance of the two
+* **Final score (F0.5)** — weights precision above recall, because false alarms erode developer trust
+
+## How claims are matched
+
+```text
+SparkLens output / ground truth
+    ↓
+Atomic claims (sentences split at "so / because / but"; pronouns resolved)
+    ↓
+Spark concept normalisation ("SMJ" = "SortMergeJoin" = "sort-merge join")
+    ↓
+TF-IDF vectors (Spark-concept channel + wording channel)
+    ↓
+Directional matching (precision: backed by the ground truth; recall: stated by SparkLens)
+    ↓
+Consistency rules (stage counts, numbers, polarity, exclusive alternatives, advice vs observation)
+```
+
+## Validating the evaluator
+
+`python evaluation/evaluate_engine.py validate` measures the evaluator itself on 1,377 test cases whose correct answer is known by construction:
+
+| Test type | Example | Correct verdict |
+|---|---|---|
+| Paraphrase | the same claim in different words | match |
+| Number change | "3 stages" → "4 stages" | no match |
+| Polarity flip | enabled ↔ disabled, required ↔ avoidable | no match |
+| Concept swap | sort-merge ↔ broadcast hash join, hash ↔ range partitioning | no match |
+| Cross-plan claim | a claim about an operator the plan text does not contain | no match |
+| Role swap | same words, roles swapped | no match |
+
+It also checks invariance (reordering, duplicating or re-spacing findings never changes a score) and bounds (the ground truth as prediction scores 1.0, an empty answer 0.0, another plan's output far lower).
+
+## Current baseline
+
+Rule engine `c7daf3c`, 4 Oct 2026:
+
+| Metric | Value |
+|---|---|
+| SparkLens precision / recall / F1 | 0.637 / 0.478 / 0.546 |
+| SparkLens final score (F0.5) | **0.597** |
+| Evaluator accuracy, 1,377 known-answer cases | **92.2%** (dev split 94.4%, test split 90.2%) |
+| Evaluator false accepts / false rejects | 11.5% / 3.5% |
+| Invariance checks / bounds | 200 of 200 / pass |
+
+Known limitation: claims that use the same words with swapped roles ("orders is broadcast, customers is shuffled" vs the reverse) are not yet told apart.
+
+## Running the evaluation
+
+```bash
+pip install -r evaluation/requirements-eval.txt
+
+python evaluation/evaluate_engine.py selftest    # unit checks of the matcher and metrics
+python evaluation/evaluate_engine.py validate    # evaluator accuracy on known-answer cases
+python evaluation/evaluate_engine.py predict     # SparkLens (--eval) on all 50 plans -> dataset_predicted.csv
+python evaluation/evaluate_engine.py evaluate    # precision / recall / F1 / final score, change vs baseline
+python evaluation/evaluate_engine.py baseline    # freeze the current result as eval_baseline.json
+```
+
+`predict` runs the SparkLens CLI, so SparkLens itself must be installed (`pip install -e .`). After a rule change, run `predict --force` and then `evaluate`; improved and regressed plans are listed.
+
+To fill `dataset_predicted.csv` by hand, run `python evaluation/evaluate_engine.py export` and paste the output of `sparklens analyze evaluation/eval_plans/<id>.txt --eval` into each row.
+
+All evaluation files live in `evaluation/`; run the commands from the repository root.
+
+| File | Purpose |
+|---|---|
+| `evaluate_engine.py` | the evaluator (commands above) |
+| `dataset_expected.csv` | ground truth |
+| `dataset_predicted.csv` | SparkLens output being scored |
+| `eval_validation.csv` | paraphrases and role swaps used by `validate` |
+| `eval_baseline.json` | frozen reference result, including evaluator accuracy |
+| `eval_review.csv` | per-claim verdicts of the last run (generated, not tracked) |
+
+---
+
+# Architecture
+
+```text
+Spark Physical Plan
+          │
+          ▼
+   Plan Normalizer
+          │
+          ▼
+      Plan Parser
+          │
+          ▼
+      DAG Builder
+          │
+          ▼
+    Execution DAG
+          │
+          ▼
+      Rule Engine
+          │
+          ▼
+   Analysis Report
+          │
+          ▼
+ Recommendation Engine
+          │
+          ▼
+     Prompt Builder
+          │
+          ▼
+      Gemini AI
+          │
+          ▼
+    AI Explanation
+
+
+Execution DAG
+      │
+      ▼
+ Visualization
+      │
+      ▼
+    dag.png
+
+
+Analysis Report
+      │
+      ▼
+ Report Generator
+      │
+      ▼
+report.md
+report.html
+```
+
+---
+
+# Installation
+
+Clone the repository:
+
+```bash
+git clone <repository-url>
+cd SparkLens
+```
+
+Install SparkLens locally:
+
+```bash
+pip install -e .
+```
+
+---
+
+# AI Setup
+
+Create a `.env` file in the project root:
+
+```env
+GEMINI_API_KEY=your_api_key_here
+```
+
+SparkLens uses Google Gemini for natural-language explanations.
+
+---
+
+# Usage
+
+## Deterministic Analysis
+
+```bash
+sparklens analyze sample_plans/broadcast_join.txt
+```
+
+Outputs:
+
+```text
+dag.png
+report.md
+report.html
+```
+
+---
+
+## AI-Enhanced Analysis
+
+```bash
+sparklens analyze sample_plans/broadcast_join.txt --ai
+```
+
+Outputs:
+
+```text
+dag.png
+report.md
+report.html
+```
+
+with AI-generated:
+
+* Executive Summary
+* Execution Explanation
+* Recommendations
+
+## Evaluation Output
+
+```bash
+sparklens analyze evaluation/eval_plans/<id>.txt --eval
+```
+
+This prints the deterministic rule-engine findings as a single line without making AI API calls.
+
+---
+
+# Example Findings
+
+```text
+Single Partition Bottleneck
+Severity: High
+
+The execution plan contains a SinglePartition exchange,
+which may reduce parallelism by processing data in a
+single partition.
+```
+
+---
+
+# Example AI Summary
+
+```text
+The execution plan uses a Broadcast Hash Join and Adaptive
+Query Execution but contains a critical Single Partition
+bottleneck that limits parallelism and may impact scalability.
+```
+
+---
+
+# Project Structure
+
+```text
+SparkLens/
+│
+├── ai/
+├── analyzer/
+├── cli/
+├── evaluation/
+├── models/
+├── report/
+├── rules/
+├── sample_plans/
+├── tests/
+├── visualization/
+│
+├── pyproject.toml
+├── README.md
+└── .env
+```
+
+---
+
+# Running Tests
+
+Execute the full test suite:
+
+```bash
+pytest
+```
+
+---
+
+# Current Capabilities (v0.1.0)
+
+* Spark Physical Plan Parsing
+* Execution DAG Construction
+* Rule-Based Analysis
+* Recommendation Generation
+* AI-Assisted Explanations
+* DAG Visualization
+* Markdown Reports
+* HTML Reports
+* CLI Interface
+* Local Package Installation
+
+---
+
+# Roadmap
+
+## v0.2
+
+* Databricks Integration
+* Rich HTML Dashboard
+* Embedded DAG Visualizations
+* Additional Rule Packs
+* Enhanced Recommendation Engine
+
+## v0.3
+
+* Runtime Metrics Analysis
+* Spark UI Integration
+* Query Plan Comparison
+* Execution Replay Visualizations
+* Cost-Based Optimization Insights
+
+---
+
+# Version
+
+Current Release:
+
+```text
+v0.1.0
+```
+
+---
+
+Built to make Spark execution plans understandable, explainable, and actionable.