dbgraph 0.1.0__py3-none-any.whl

dbgraph-0.1.0.dist-info/METADATA

@@ -0,0 +1,165 @@
+Metadata-Version: 2.5
+Name: dbgraph
+Version: 0.1.0
+Summary: dbgraph is a database profiling tool which model the database as a graph
+Requires-Python: >=3.11
+Requires-Dist: bm25s[all]>=0.3.9
+Requires-Dist: build>=1.6.0
+Requires-Dist: grip>=4.6.2
+Requires-Dist: langchain-openai>=1.6.0
+Requires-Dist: langchain>=1.3.16
+Requires-Dist: matplotlib>=3.11.0
+Requires-Dist: mkdocs-gen-files>=0.6.1
+Requires-Dist: mkdocs-material>=9.7.7
+Requires-Dist: mkdocs>=1.6.1
+Requires-Dist: mkdocstrings[python]>=1.0.6
+Requires-Dist: openai>=2.45.0
+Requires-Dist: psutil>=7.2.2
+Requires-Dist: pydantic>=2.13.4
+Requires-Dist: pydot>=4.0.1
+Requires-Dist: pympler>=1.1
+Requires-Dist: pystemmer>=3.1.0
+Requires-Dist: pytest>=9.1.1
+Requires-Dist: python-dotenv>=1.2.2
+Requires-Dist: rustworkx>=0.18.0
+Requires-Dist: sqlalchemy>=2.0.52
+Requires-Dist: tqdm>=4.68.4
+Requires-Dist: trino>=0.339.0
+Description-Content-Type: text/markdown
+
+# DBGraph - Building schema graph with LLM Assistance
+
+**DBGraph** aims to help data scientist with exploring and finding relevant data assets in a huge and complex database.
+
+## Quick Start
+
+Here is a quick example of how to build and query a schema graph with DBGraph:
+
+```python
+# imports
+from dbgraph.builder.sqlite.sqlite_graph_builder import SQLiteGraphBuilder
+from dbgraph.io.json_graph_writer import JSONGraphWriter
+from dbgraph.io.json_graph_loader import JSONGraphLoader
+from dbgraph.search.bm25_search_engine import BM25SearchEngine
+
+# initiate GraphBuilder on an SQLite database
+graph_builder = SQLiteGraphBuilder(Path("data/northwind.db"))
+
+# build the schema graph (include profiling databases)
+graph = graph_builder.build_graph()
+
+# generate descriptions for assets
+graph_descriptor = GraphDescriptorV1(
+    llm=OAICompatibleLLM(
+        model=...,
+        base_url=...,
+        api_key=...,
+    ),
+    system_prompt=...,
+    formating_prompt=...,
+    target_prompt=...
+)
+graph = graph_descriptor.rfill_semantic_aspects(graph)
+
+# save the graph
+graph_writer = JSONGraphWriter(
+    json_path=Path("data/northwind-graph.json"), indent=2
+)
+graph_writer.write()
+
+# load the graph
+graph_loader = JSONGraphLoader(json_path=Path("data/northwind-graph.json"))
+graph = self.graph_loader.load()
+
+# index the graph using BM25
+search_engine = BM25SearchEngine(Path("data/northwind-index"))
+semantic_aspects = {
+    a.asset_id: cast(SemanticAspect, a.aspects["semantic_properties"])
+    for a in graph.assets
+}
+search_engine.index(semantic_aspects)
+
+# retrieve assets using BM25
+assets_ids = search_engine.search(
+    "Give me the total count of orders in each categories"
+)
+```
+
+For visualization purpose, this is the graph saved in JSON:
+
+```json
+{
+  "assets": [
+    {
+      "asset_id": "8ab5a624-0596-497e-a0ee-3996d95dbe63",
+      "name": "Categories",
+      "type": "table",
+      "aspects": {
+        "schema_properties": {
+          "name": "Categories_table_schema",
+          "pks": ["CategoryID"],
+          "indices": {}
+        },
+        "statistical_properties": {
+          "name": "Categories_table_stats",
+          "num_columns": 4,
+          "num_rows": 8
+        },
+        "semantic_properties": {
+          "name": "Categories_semantic",
+          "description": "Stores product category definitions and metadata, serving as a lookup table for classifying products in the inventory system.",
+          "keywords": [
+            "categories",
+            "product classification",
+            "category definitions",
+            "inventory groups",
+            "product types"
+          ]
+        }
+      }
+    },
+    ...
+  ],
+  "links": [
+    {
+      "link_id": "db6bea93-a02c-4426-a2db-449e4a7bba8f",
+      "name": "Categories_CategoryID",
+      "type": "contain",
+      "source_id": "8ab5a624-0596-497e-a0ee-3996d95dbe63",
+      "destination_id": "04c20046-2808-4021-bbf1-99876e0eea6e",
+      "aspects": {}
+    },
+    ...
+  ]
+}
+```
+
+## Usecases
+
+![Usecases of DBGraph](diagrams/usecase.png)
+There are 4 main groups of usecases where DBGraph is applicable:
+
+- **Manipulating database schema**: Build the schema graph, store it, and use it to traverse around the database, find JOIN path, get references tables, ...
+- **Profiling database**: Use the concept of `Aspect` to represent different types of properties attached to a single data asset.
+  Each data asset can have a statistics aspects, semantical aspects, ...
+- **Render graph**: Output schema graph to Markdown or text as context for LLM
+- **LLM Assistance**: Leverage LLM to generate data assets' descriptions and tags. Furthermore, LLM could also be used in the process of SQL generation.
+- **Search for data assets**: Search for wanted data assets based on their descriptions. The descriptions are indexed and retrieved with BM25 algorithms.
+
+## Architecture
+
+![Class diagram of DBGraph](diagrams/entity.png)
+
+To encourage open-ness and extension, DBGraph is designed in a way that is very easy to extend.
+
+1. **The core classes** (_entities_) define the shared business logic of database graphs (traversal, neighborhoods, ...) and core operations
+   within the application (building graphs, profiling databases, ...). The prefix _"R..."_ stands for _"Relational"_, as the class is dedicated
+   Relational databases only. The same stands for _"D.."_ (Document), _"V..."_ (Vector), _"G..."_ (Graph). However, at this point, the only supported
+   paradigm is Relational database.
+2. **The interfaces** (_extensions_) define a part of the system that should be **pluggable**. For example:
+
+   - `RProfiler` and `RGraphBuilder` should works on multiple types of RDBMS, not just SQLite. Hence the abstraction of `RDataGateway`.
+   - We want to support multiple LLM providers. Hence the abstraction of `LLM`.
+   - There are many ways to render a graph into Markdown or text. Hence the abstraction of `RGraphRenderer`.
+   - To store and load the graph's data from some storages, we have multiple options. Hence the abstraction of `GraphGateway`.
+   - To index and retrieve graph's data from some search engines, we also have multiple options. Hence the abstraction of `SearchEngine`

dbgraph-0.1.0.dist-info/RECORD

@@ -0,0 +1,42 @@
+dbgraph/__init__.py,sha256=_4z_ceG0sw9nMyaxMxzdcenmtVjwiAgpDDYacYLfp48,716
+dbgraph/application.py,sha256=53L9KYLAfS9LkNY8OPZqMZuJ54RWWSreZRu2otrDkh8,2748
+dbgraph/builder/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+dbgraph/builder/graph_builder.py,sha256=QlR_LB_WIYELX4y1IuztdbjWr3BeO_TgTielqAGNiDo,586
+dbgraph/builder/sql_graph_builder.py,sha256=vxXbLQ5G_etbQ-vc8Q4_Ot0EuMua3AoGRJZPRG3tnwc,9034
+dbgraph/descriptor/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+dbgraph/descriptor/graph_descriptor.py,sha256=pz8ED3mhJcr_zCY8wm2-S8mAU_-PWoO_0rl_Xy5BX0c,2635
+dbgraph/descriptor/langchain_graph_descriptor.py,sha256=UxiYIj-k5yGgvMMNPviy-Vn1IY0xVP5OtaOGCRnDjXA,2810
+dbgraph/descriptor/prompt_templates.py,sha256=rMMgdquBnIy93qjgbK-X2t6h-b0dyM_Wuh0OUJ-D6wo,4000
+dbgraph/entity/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+dbgraph/entity/aspect.py,sha256=U2ytORdxGo7D9UmtxCkAQmudUupylmgsVQEytdOK5gY,3904
+dbgraph/entity/asset.py,sha256=5fu7aSAbIuiqWEcwb07D3YiwGUmVLrPCfWmaIuyEJdc,627
+dbgraph/entity/asset_type.py,sha256=kXoNXkOY0PQ64EgVFvJSzrgBFnkBIH_cSXQT3lNLRbk,238
+dbgraph/entity/dbgraph.py,sha256=f8iKvBQCrDeBMddLzMe5iAPFDLGGgJl3Jjv8oB3FTQQ,5356
+dbgraph/entity/link.py,sha256=Op8IKqqFUAGtinXV8u8DN0Zo_x6hoQ8XoB-YwTJxnp8,1185
+dbgraph/entity/link_type.py,sha256=VYWI2ImncHi39Dm8vZqp3yiAnshTSuZVnyaCt3BJ42o,272
+dbgraph/entity/rdbgraph.py,sha256=fl717kbMdvFVsby5Kbk9mq9u8ZCS3G-8-vNhbxU0fZ8,2759
+dbgraph/event/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+dbgraph/event/event.py,sha256=xELHK_r7qZzigX_dB4B8YKzLXd8ZuDOz9PB8RRzSCCg,386
+dbgraph/event/event_bus.py,sha256=Aj4_drzpHT6DW9AbsJvcLF5BZu-8uvQV6wCFj337tYo,2087
+dbgraph/event/event_handler.py,sha256=x5R5BFfJ8Qon1aZusyKdJ79imlY7xLCXJ5tp5VltpDw,239
+dbgraph/io/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+dbgraph/io/graph_loader.py,sha256=53ROOIZYoSIuWy1ALzsMUySm-WeahumqxdfrhWVS7I0,253
+dbgraph/io/graph_writer.py,sha256=nWJ3xe97LXKOJm763R7h-lreiCGGrSIudhsgbLCGNBQ,257
+dbgraph/io/json_graph_loader.py,sha256=GoV_ziCpPCIO_TAK68_iMUcI180O8Mv1Ad2iqwciQSQ,6270
+dbgraph/io/json_graph_writer.py,sha256=kmKMHfOA8ieJZgJck3gkJ1F2YCjVXEfUNojwIw4dD8k,473
+dbgraph/persistent/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+dbgraph/persistent/graph_persistent.py,sha256=9itKwW-OSWtfx7TXdWtl-CrfCaKphMVkjfxkh7VX2Vc,3298
+dbgraph/persistent/models.py,sha256=wjXnPDbQqMtMO_uhASfGFBRkjzIsQ8QXQd5FBOPn7F4,4900
+dbgraph/persistent/sql_graph_persistent.py,sha256=rrLCEKN0jzseC_iPG2nPHeBHv6ib0Ihyx2DbgUswbQ8,15151
+dbgraph/render/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+dbgraph/render/graph_renderer.py,sha256=SJvf0IZGDnEjy9n2uXD5mUq7FIm5nL_Gw8k2cn15i5E,294
+dbgraph/render/markdown_renderer.py,sha256=oJKRf7JNZUEEeiS-IUrxq6mbkWq3PAXXgzBSqe-yOR0,4846
+dbgraph/render/text_renderer.py,sha256=l0Q8zh-tN09UVqSpNNrS9XJ9MRPqmryYS3eixdM1eZ4,289
+dbgraph/search/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+dbgraph/search/bm25_search_engine.py,sha256=xayZSIMSgGjbSkI1jsT7tC6S2wvQ4mkOd39OA36bdHk,1168
+dbgraph/search/search_engine.py,sha256=sXnA6VOmd_pKuBwhXrBw38Tx7c10lFJi2NQdLnNCCa0,414
+dbgraph/utils/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+dbgraph/utils/singleton.py,sha256=9sYBaQt3ILqoMgovYVusMqSzmi60yA6VbMFRiw78LLs,1498
+dbgraph-0.1.0.dist-info/METADATA,sha256=DqCWa8dQp23myPTy9po9Qh-r9xQzor5ZhfZHe_Q2jQo,6073
+dbgraph-0.1.0.dist-info/WHEEL,sha256=zOwg4jB6zX2kU910N-cMawjivD6tO8NEWvE12je1bVk,87
+dbgraph-0.1.0.dist-info/RECORD,,

dbgraph-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.32.0
+Root-Is-Purelib: true
+Tag: py3-none-any