bills-sql-lineage 0.0.1__tar.gz

bills_sql_lineage-0.0.1/PKG-INFO

@@ -0,0 +1,80 @@
+Metadata-Version: 2.1
+Name: bills-sql-lineage
+Version: 0.0.1
+Summary: Personal SQL lineage generator.
+Author: Bilbottom
+Requires-Python: >=3.11,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: arguably (>=1.2.5,<2.0.0)
+Requires-Dist: sqlglot[rs] (>=23.0.5,<24.0.0)
+Description-Content-Type: text/markdown
+
+<div align="center">
+
+[![Python](https://img.shields.io/badge/Python-3.11+-blue.svg)](https://www.python.org/downloads/release/python-3110/)
+[![Poetry](https://img.shields.io/endpoint?url=https://python-poetry.org/badge/v0.json)](https://python-poetry.org/)
+[![tests](https://github.com/Bilbottom/sql-lineage/actions/workflows/tests.yaml/badge.svg)](https://github.com/Bilbottom/sql-lineage/actions/workflows/tests.yaml)
+[![coverage](coverage.svg)](https://github.com/dbrgn/coverage-badge)
+![GitHub last commit](https://img.shields.io/github/last-commit/Bilbottom/sql-lineage)
+
+[![code style: prettier](https://img.shields.io/badge/code_style-prettier-ff69b4.svg?style=flat-square)](https://github.com/prettier/prettier)
+[![code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
+[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/Bilbottom/sql-lineage/main.svg)](https://results.pre-commit.ci/latest/github/Bilbottom/sql-lineage/main)
+[![Sourcery](https://img.shields.io/badge/Sourcery-enabled-brightgreen)](https://sourcery.ai)
+
+</div>
+
+---
+
+# SQL Lineage 🔀
+
+Personal SQL lineage generator.
+
+Built on the following awesome libraries:
+
+- [SQLGlot](https://github.com/tobymao/sqlglot) for SQL parsing
+- [arguably](https://github.com/treykeown/arguably) for the CLI
+
+## So... what is this? 🤔
+
+> [!NOTE]
+>
+> This project is still in development and currently only parses the CTEs of an SQL query since that's all I need for now.
+
+I write _a lot_ of SQL, and I often need to understand "lineage" in lots of different ways.
+
+Things like [dbt](https://www.getdbt.com/) and [SQLMesh](https://sqlmesh.com/) are great for object lineage (tables, columns, etc.), but I often need to understand "lineage" like the CTE lineage in a query, among other things.
+
+This project is just a personal tool to help me generate lineage diagrams (in [Mermaid](https://mermaid.js.org/) syntax) for SQL queries to fit that requirement.
+
+Upcoming improvements will (hopefully) include:
+
+- Column lineage to Mermaid
+- Semantic edges (e.g. distinguish between `JOIN`, `WHERE`, `UNION`, etc.)
+- Lineage for multiple files in a single diagram
+
+## Installation ⬇️
+
+Grab a copy from PyPI like usual (note the `bills-` prefix):
+
+```
+pip install bills-sql-lineage
+```
+
+## Usage 📖
+
+Pass the path to a SQL file to the `lineage` command to generate the lineage as a [Mermaid](https://mermaid.js.org/) diagram:
+
+```
+lineage path/to/file.sql
+```
+
+This will write a Mermaid diagram to `path/to/file.mermaid`. You can control the target with the second argument:
+
+```
+lineage path/to/file.sql path/to/output.mermaid
+```
+

bills_sql_lineage-0.0.1/README.md

@@ -0,0 +1,66 @@
+<div align="center">
+
+[![Python](https://img.shields.io/badge/Python-3.11+-blue.svg)](https://www.python.org/downloads/release/python-3110/)
+[![Poetry](https://img.shields.io/endpoint?url=https://python-poetry.org/badge/v0.json)](https://python-poetry.org/)
+[![tests](https://github.com/Bilbottom/sql-lineage/actions/workflows/tests.yaml/badge.svg)](https://github.com/Bilbottom/sql-lineage/actions/workflows/tests.yaml)
+[![coverage](coverage.svg)](https://github.com/dbrgn/coverage-badge)
+![GitHub last commit](https://img.shields.io/github/last-commit/Bilbottom/sql-lineage)
+
+[![code style: prettier](https://img.shields.io/badge/code_style-prettier-ff69b4.svg?style=flat-square)](https://github.com/prettier/prettier)
+[![code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
+[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/Bilbottom/sql-lineage/main.svg)](https://results.pre-commit.ci/latest/github/Bilbottom/sql-lineage/main)
+[![Sourcery](https://img.shields.io/badge/Sourcery-enabled-brightgreen)](https://sourcery.ai)
+
+</div>
+
+---
+
+# SQL Lineage 🔀
+
+Personal SQL lineage generator.
+
+Built on the following awesome libraries:
+
+- [SQLGlot](https://github.com/tobymao/sqlglot) for SQL parsing
+- [arguably](https://github.com/treykeown/arguably) for the CLI
+
+## So... what is this? 🤔
+
+> [!NOTE]
+>
+> This project is still in development and currently only parses the CTEs of an SQL query since that's all I need for now.
+
+I write _a lot_ of SQL, and I often need to understand "lineage" in lots of different ways.
+
+Things like [dbt](https://www.getdbt.com/) and [SQLMesh](https://sqlmesh.com/) are great for object lineage (tables, columns, etc.), but I often need to understand "lineage" like the CTE lineage in a query, among other things.
+
+This project is just a personal tool to help me generate lineage diagrams (in [Mermaid](https://mermaid.js.org/) syntax) for SQL queries to fit that requirement.
+
+Upcoming improvements will (hopefully) include:
+
+- Column lineage to Mermaid
+- Semantic edges (e.g. distinguish between `JOIN`, `WHERE`, `UNION`, etc.)
+- Lineage for multiple files in a single diagram
+
+## Installation ⬇️
+
+Grab a copy from PyPI like usual (note the `bills-` prefix):
+
+```
+pip install bills-sql-lineage
+```
+
+## Usage 📖
+
+Pass the path to a SQL file to the `lineage` command to generate the lineage as a [Mermaid](https://mermaid.js.org/) diagram:
+
+```
+lineage path/to/file.sql
+```
+
+This will write a Mermaid diagram to `path/to/file.mermaid`. You can control the target with the second argument:
+
+```
+lineage path/to/file.sql path/to/output.mermaid
+```

bills_sql_lineage-0.0.1/pyproject.toml

@@ -0,0 +1,58 @@
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"
+
+
+[tool.poetry]
+name = "bills-sql-lineage"  # For PyPI
+version = "0.0.1"
+description = "Personal SQL lineage generator."
+authors = ["Bilbottom"]
+readme = "README.md"
+packages = [{include = "sql_lineage"}]
+
+[tool.poetry.scripts]
+lineage = 'sql_lineage.__main__:app'
+
+[tool.poetry.dependencies]
+python = "^3.11"
+sqlglot = {extras = ["rs"], version = "^23.0.5"}
+arguably = "^1.2.5"
+
+[tool.poetry.group]
+dev.optional = true
+test.optional = true
+ide.optional = true
+
+[tool.poetry.group.dev.dependencies]
+coverage-badge = "^1.1.0"
+pre-commit = "^3.6.2"
+pylint = "3.1.0"
+
+[tool.poetry.group.test.dependencies]
+pytest = "^8.0.2"
+pytest-cov = "^4.1.0"
+
+# Packages just for IDE integration
+[tool.poetry.group.ide.dependencies]
+black = "*"
+
+
+[tool.pytest.ini_options]
+addopts = "--cov=sql_lineage --cov-fail-under=80"
+testpaths = ["tests"]
+
+
+[tool.coverage.run]
+omit = ["__main__.py"]
+
+
+[tool.isort]
+profile = "black"
+
+
+[tool.pylint.format]
+max-line-length = 120
+
+[tool.pylint.MASTER]
+ignore-paths = "^tests/.*$"

bills_sql_lineage-0.0.1/sql_lineage/__main__.py

@@ -0,0 +1,12 @@
+"""
+CLI entry point for the ``sql_lineage`` package.
+"""
+
+import arguably
+
+# This import is necessary for the CLI to work
+import sql_lineage.main  # pylint: disable=unused-import
+
+app = arguably.run  # sourcery skip: avoid-global-variables
+if __name__ == "__main__":
+    app()

bills_sql_lineage-0.0.1/sql_lineage/lineage.py

@@ -0,0 +1,55 @@
+"""
+Parsers for SQL queries and Mermaid graphs.
+"""
+
+from __future__ import annotations
+
+import sqlglot
+from sqlglot import exp
+from sqlglot.optimizer import qualify
+
+Nodes = list[str]
+Edges = list[tuple[str, str]]  # Directed edges, source --> target.
+
+
+def _parse_ctes(sql: str) -> [Nodes, Edges]:
+    """
+    Parse an SQL query into nodes and edges based on CTEs.
+    """
+    parsed = qualify.qualify(sqlglot.parse_one(sql))
+    nodes, edges = [], []
+
+    # Lazy and error-prone CTE sourcing
+    for cte in parsed.find_all(exp.CTE):
+        nodes.append(cte.alias)
+        edges.extend((tbl.name, cte.alias) for tbl in cte.find_all(exp.Table))
+
+    parsed.args["with"] = None  # Dirty hack to remove the CTEs from the query
+    nodes.append("final")
+    edges.extend((tbl.name, "final") for tbl in parsed.find_all(exp.Table))
+
+    return nodes, edges
+
+
+def _to_mermaid(nodes: Nodes, edges: Edges) -> str:
+    """
+    Convert nodes and edges into a Mermaid graph.
+    """
+    mermaid = "graph TD\n"
+
+    for node in nodes:
+        mermaid += f"    {node}\n"  # pylint: disable=consider-using-join
+
+    for edge in edges:
+        mermaid += f"    {edge[0]} --> {edge[1]}\n"
+
+    return mermaid
+
+
+def sql_to_mermaid(sql: str) -> str:
+    """
+    Convert an SQL query into a Mermaid graph.
+    """
+    nodes, edges = _parse_ctes(sql)
+
+    return _to_mermaid(nodes, edges)

bills_sql_lineage-0.0.1/sql_lineage/main.py

@@ -0,0 +1,41 @@
+"""
+An SQL lineage tool, built on the awesome SQLGlot library.
+"""
+
+from __future__ import annotations
+
+import pathlib
+
+import arguably
+
+from sql_lineage import lineage
+
+
+def _to_path(file_path: str, validate: bool = False) -> pathlib.Path:
+    """
+    Convert a file path to a pathlib.Path object.
+    """
+    path = pathlib.Path(file_path)
+    if validate:
+        if not path.exists():
+            raise FileNotFoundError(f"File not found: {file_path}")
+        if not path.is_file():
+            raise ValueError(f"Not a file: {file_path}")
+
+    return path
+
+
+@arguably.command
+def main(file: str, target: str = None) -> None:
+    """
+    Parse an SQL file into a Mermaid graph by tracing CTEs.
+    """
+    file_path = _to_path(file, validate=True)
+    target_path = _to_path(target) if target else file_path.with_suffix(".mermaid")
+
+    sql = file_path.read_text(encoding="utf-8")
+    mermaid = lineage.sql_to_mermaid(sql)
+
+    target_path.write_text(mermaid, encoding="utf-8")
+    # Switch to logging at some point
+    print(f"Mermaid graph written to: {target_path.absolute()}")