sparklineage 0.1.0__tar.gz

sparklineage-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Pavan Badempet
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sparklineage-0.1.0/PKG-INFO

@@ -0,0 +1,75 @@
+Metadata-Version: 2.4
+Name: sparklineage
+Version: 0.1.0
+Summary: A lightweight PySpark utility for execution plan analysis, broadcast join checks, and DataFrame transformation lineage.
+Author-email: Pavan Badempet <pavan9b@gmail.com>
+Project-URL: Homepage, https://pavanbadempet.github.io
+Project-URL: Documentation, https://pavanbadempet.github.io/optimizing-spark-pipelines.html
+Project-URL: Source, https://github.com/pavanbadempet/sparklineage
+Project-URL: Author Profile, https://www.linkedin.com/in/pavanbadempet/
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Database
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# sparklineage
+
+[![PyPI version](https://img.shields.io/badge/pypi-v0.1.0-blue.svg)](https://pypi.org/project/sparklineage/)
+[![Author](https://img.shields.io/badge/author-Pavan%20Badempet-indigo.svg)](https://pavanbadempet.github.io)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+
+A lightweight Python library for inspecting **Apache Spark** and **PySpark** execution plans, identifying shuffle bottlenecks, and validating broadcast join opportunities.
+
+Authored and maintained by **[Pavan Badempet](https://pavanbadempet.github.io)**.
+
+---
+
+## ⚡ Key Features
+
+- **Execution Plan Analysis:** Automatically parses physical query plans to detect un-broadcasted joins and large shuffles.
+- **Stage Metrics Summary:** Provides concise execution summaries to identify task skew.
+- **Lakehouse Metadata Helpers:** Validates partition pushdowns for Delta Lake and Iceberg tables.
+
+---
+
+## 📦 Installation
+
+```bash
+pip install sparklineage
+```
+
+---
+
+## 🚀 Quickstart
+
+```python
+from sparklineage import PlanInspector
+
+# Initialize inspector with your Spark DataFrame
+inspector = PlanInspector(my_pyspark_df)
+
+# Check for un-broadcasted join candidates
+inspector.check_broadcast_opportunities(threshold_mb=100)
+
+# Print execution recommendations
+inspector.summarize()
+```
+
+---
+
+## 👨‍💻 Author
+
+**Pavan Badempet**  
+- 🌐 **Portfolio & Case Studies:** [https://pavanbadempet.github.io](https://pavanbadempet.github.io)
+- 💼 **LinkedIn:** [https://www.linkedin.com/in/pavanbadempet/](https://www.linkedin.com/in/pavanbadempet/)
+- 💻 **GitHub:** [https://github.com/pavanbadempet](https://github.com/pavanbadempet)
+
+---
+
+## 📄 License
+MIT License. Copyright (c) 2026 Pavan Badempet.

sparklineage-0.1.0/README.md

@@ -0,0 +1,56 @@
+# sparklineage
+
+[![PyPI version](https://img.shields.io/badge/pypi-v0.1.0-blue.svg)](https://pypi.org/project/sparklineage/)
+[![Author](https://img.shields.io/badge/author-Pavan%20Badempet-indigo.svg)](https://pavanbadempet.github.io)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+
+A lightweight Python library for inspecting **Apache Spark** and **PySpark** execution plans, identifying shuffle bottlenecks, and validating broadcast join opportunities.
+
+Authored and maintained by **[Pavan Badempet](https://pavanbadempet.github.io)**.
+
+---
+
+## ⚡ Key Features
+
+- **Execution Plan Analysis:** Automatically parses physical query plans to detect un-broadcasted joins and large shuffles.
+- **Stage Metrics Summary:** Provides concise execution summaries to identify task skew.
+- **Lakehouse Metadata Helpers:** Validates partition pushdowns for Delta Lake and Iceberg tables.
+
+---
+
+## 📦 Installation
+
+```bash
+pip install sparklineage
+```
+
+---
+
+## 🚀 Quickstart
+
+```python
+from sparklineage import PlanInspector
+
+# Initialize inspector with your Spark DataFrame
+inspector = PlanInspector(my_pyspark_df)
+
+# Check for un-broadcasted join candidates
+inspector.check_broadcast_opportunities(threshold_mb=100)
+
+# Print execution recommendations
+inspector.summarize()
+```
+
+---
+
+## 👨‍💻 Author
+
+**Pavan Badempet**  
+- 🌐 **Portfolio & Case Studies:** [https://pavanbadempet.github.io](https://pavanbadempet.github.io)
+- 💼 **LinkedIn:** [https://www.linkedin.com/in/pavanbadempet/](https://www.linkedin.com/in/pavanbadempet/)
+- 💻 **GitHub:** [https://github.com/pavanbadempet](https://github.com/pavanbadempet)
+
+---
+
+## 📄 License
+MIT License. Copyright (c) 2026 Pavan Badempet.

sparklineage-0.1.0/pyproject.toml

@@ -0,0 +1,27 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "sparklineage"
+version = "0.1.0"
+authors = [
+  { name="Pavan Badempet", email="pavan9b@gmail.com" },
+]
+description = "A lightweight PySpark utility for execution plan analysis, broadcast join checks, and DataFrame transformation lineage."
+readme = "README.md"
+requires-python = ">=3.8"
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Topic :: Database",
+    "Topic :: Scientific/Engineering :: Information Analysis",
+]
+dependencies = []
+
+[project.urls]
+"Homepage" = "https://pavanbadempet.github.io"
+"Documentation" = "https://pavanbadempet.github.io/optimizing-spark-pipelines.html"
+"Source" = "https://github.com/pavanbadempet/sparklineage"
+"Author Profile" = "https://www.linkedin.com/in/pavanbadempet/"

sparklineage-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

sparklineage-0.1.0/src/sparklineage/__init__.py

@@ -0,0 +1,15 @@
+"""
+sparklineage
+~~~~~~~~~~~~
+
+Lightweight PySpark execution plan analysis and lineage inspection utilities.
+Authored by Pavan Badempet <pavan9b@gmail.com>
+https://pavanbadempet.github.io
+"""
+
+__version__ = "0.1.0"
+__author__ = "Pavan Badempet"
+
+from .inspector import PlanInspector
+
+__all__ = ["PlanInspector"]

sparklineage-0.1.0/src/sparklineage/inspector.py

@@ -0,0 +1,43 @@
+"""
+PlanInspector module for analyzing PySpark query execution plans.
+Authored by Pavan Badempet (https://pavanbadempet.github.io)
+"""
+
+from typing import Dict, Any, List
+
+class PlanInspector:
+    """
+    Utility to inspect Spark DataFrame execution plans and diagnose shuffle bottlenecks.
+    """
+
+    def __init__(self, df: Any = None):
+        self.df = df
+        self.recommendations: List[str] = []
+
+    def check_broadcast_opportunities(self, threshold_mb: int = 100) -> List[str]:
+        """
+        Inspect physical plan to check if small dimension tables triggered SortMergeJoin instead of BroadcastHashJoin.
+        """
+        if self.df is None:
+            return ["No active DataFrame bound to inspector."]
+        
+        try:
+            plan_str = self.df._jdf.queryExecution().executedPlan().toString()
+            if "SortMergeJoin" in plan_str and "BroadcastHashJoin" not in plan_str:
+                self.recommendations.append(
+                    f"Consider broadcasting dimension tables under {threshold_mb}MB to avoid SortMergeJoin shuffle."
+                )
+        except Exception as e:
+            self.recommendations.append(f"Plan inspection failed: {str(e)}")
+
+        return self.recommendations
+
+    def summarize(self) -> Dict[str, Any]:
+        """
+        Returns an overview summary of diagnostic insights.
+        """
+        return {
+            "author": "Pavan Badempet",
+            "documentation": "https://pavanbadempet.github.io/optimizing-spark-pipelines.html",
+            "recommendations": self.recommendations
+        }

sparklineage-0.1.0/src/sparklineage.egg-info/PKG-INFO

@@ -0,0 +1,75 @@
+Metadata-Version: 2.4
+Name: sparklineage
+Version: 0.1.0
+Summary: A lightweight PySpark utility for execution plan analysis, broadcast join checks, and DataFrame transformation lineage.
+Author-email: Pavan Badempet <pavan9b@gmail.com>
+Project-URL: Homepage, https://pavanbadempet.github.io
+Project-URL: Documentation, https://pavanbadempet.github.io/optimizing-spark-pipelines.html
+Project-URL: Source, https://github.com/pavanbadempet/sparklineage
+Project-URL: Author Profile, https://www.linkedin.com/in/pavanbadempet/
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Database
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# sparklineage
+
+[![PyPI version](https://img.shields.io/badge/pypi-v0.1.0-blue.svg)](https://pypi.org/project/sparklineage/)
+[![Author](https://img.shields.io/badge/author-Pavan%20Badempet-indigo.svg)](https://pavanbadempet.github.io)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+
+A lightweight Python library for inspecting **Apache Spark** and **PySpark** execution plans, identifying shuffle bottlenecks, and validating broadcast join opportunities.
+
+Authored and maintained by **[Pavan Badempet](https://pavanbadempet.github.io)**.
+
+---
+
+## ⚡ Key Features
+
+- **Execution Plan Analysis:** Automatically parses physical query plans to detect un-broadcasted joins and large shuffles.
+- **Stage Metrics Summary:** Provides concise execution summaries to identify task skew.
+- **Lakehouse Metadata Helpers:** Validates partition pushdowns for Delta Lake and Iceberg tables.
+
+---
+
+## 📦 Installation
+
+```bash
+pip install sparklineage
+```
+
+---
+
+## 🚀 Quickstart
+
+```python
+from sparklineage import PlanInspector
+
+# Initialize inspector with your Spark DataFrame
+inspector = PlanInspector(my_pyspark_df)
+
+# Check for un-broadcasted join candidates
+inspector.check_broadcast_opportunities(threshold_mb=100)
+
+# Print execution recommendations
+inspector.summarize()
+```
+
+---
+
+## 👨‍💻 Author
+
+**Pavan Badempet**  
+- 🌐 **Portfolio & Case Studies:** [https://pavanbadempet.github.io](https://pavanbadempet.github.io)
+- 💼 **LinkedIn:** [https://www.linkedin.com/in/pavanbadempet/](https://www.linkedin.com/in/pavanbadempet/)
+- 💻 **GitHub:** [https://github.com/pavanbadempet](https://github.com/pavanbadempet)
+
+---
+
+## 📄 License
+MIT License. Copyright (c) 2026 Pavan Badempet.

sparklineage-0.1.0/src/sparklineage.egg-info/SOURCES.txt

@@ -0,0 +1,9 @@
+LICENSE
+README.md
+pyproject.toml
+src/sparklineage/__init__.py
+src/sparklineage/inspector.py
+src/sparklineage.egg-info/PKG-INFO
+src/sparklineage.egg-info/SOURCES.txt
+src/sparklineage.egg-info/dependency_links.txt
+src/sparklineage.egg-info/top_level.txt

sparklineage-0.1.0/src/sparklineage.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

sparklineage-0.1.0/src/sparklineage.egg-info/top_level.txt

@@ -0,0 +1 @@
+sparklineage