cpg2py 1.0.5__py3-none-any.whl → 1.1.0__py3-none-any.whl

cpg2py-1.1.0.dist-info/METADATA

@@ -0,0 +1,407 @@
+Metadata-Version: 2.1
+Name: cpg2py
+Version: 1.1.0
+Summary: A graph-based data structure designed for querying CSV files in Joern format in Python
+Author-email: samhsu-dev <yxu166@jhu.edu>
+License: MIT License
+        
+        Copyright (c) 2025 Yichao Xu
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Homepage, https://github.com/YichaoXu/cpg2py
+Project-URL: Repository, https://github.com/YichaoXu/cpg2py
+Project-URL: Documentation, https://github.com/YichaoXu/cpg2py
+Keywords: Joern,CPG,Graph,CSV
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
+Provides-Extra: test
+
+# **cpg2py: Graph-Based Query Engine for Joern CSV Files**
+
+`cpg2py` is a Python library that provides a lightweight **graph-based query engine** for analyzing **Code Property Graphs (CPG)** extracted from Joern CSV files. The library offers an **abstract base class (ABC) architecture**, allowing users to extend and implement their own custom graph queries.
+
+---
+
+## **🚀 Features**
+
+- **MultiDiGraph Representation**: A directed multi-graph with support for multiple edges between nodes.
+- **CSV-Based Graph Construction**: Reads `nodes.csv` and `rels.csv` to construct a graph structure.
+- **Type-Safe Generic Types**: Uses Python generics for type-safe graph operations (similar to Java generics).
+- **Extensible Abstract Base Classes (ABC)**:
+  - `AbcGraphQuerier` for implementing **custom graph queries** with generic type support.
+  - `AbcNodeQuerier` for interacting with **nodes**.
+  - `AbcEdgeQuerier` for interacting with **edges**.
+- **Built-in Query Mechanisms**:
+  - **Retrieve all nodes and edges** with type-safe iteration.
+  - **Get incoming and outgoing edges** of a node.
+  - **Find successors and predecessors** with type preservation.
+  - **Traverse AST, Control Flow, and Data Flow Graphs**.
+- **Concrete Implementation**: `CpgGraph`, `CpgNode`, and `CpgEdge` provide ready-to-use implementations.
+
+---
+
+## **📚 Installation**
+
+### Using pip
+
+To install the package, use:
+
+```bash
+pip install git+https://github.com/samhsu-dev/cpg2py.git
+```
+
+### Using uv (Recommended)
+
+This project uses [uv](https://github.com/astral-sh/uv) for fast and reliable package management.
+
+**Install uv:**
+```bash
+curl -LsSf https://astral.sh/uv/install.sh | sh
+```
+
+**Clone and install:**
+```bash
+git clone https://github.com/samhsu-dev/cpg2py.git
+cd cpg2py
+uv sync --dev  # Install with dev dependencies
+```
+
+**For development:**
+```bash
+uv sync --dev
+uv run pytest tests/  # Run tests
+```
+
+Or clone the pip repository:
+
+```bash
+pip install cpg2py
+```
+
+---
+
+## **📂 File Structure**
+
+- **`nodes.csv`** (Example):
+```csv
+id:int	labels:label	type	flags:string_array	lineno:int	code	childnum:int	funcid:int	classname	namespace	endlineno:int	name	doccomment
+0	Filesystem	Directory									"input"	
+1	Filesystem	File									"example.php"	
+2	AST	AST_TOPLEVEL	TOPLEVEL_FILE	1					""	25	"/input/example.php"	
+
+````
+- **`rels.csv`** (Example):
+```csv
+start	end	type
+2	3	ENTRY
+2	4	EXIT
+6	7	ENTRY
+6	9	PARENT_OF
+````
+
+---
+
+## **🎯 Type Safety with Generics**
+
+`cpg2py` uses Python's generic types (similar to Java generics) to provide type-safe operations:
+
+```python
+from typing import Iterable
+from cpg2py import cpg_graph, CpgGraph, CpgNode, CpgEdge
+
+# Type checker knows graph is CpgGraph[CpgNode, CpgEdge]
+graph: CpgGraph = cpg_graph("nodes.csv", "rels.csv")
+
+# Type checker knows node is CpgNode (not just AbcNodeQuerier)
+node: CpgNode = graph.node("5")
+
+# Type checker knows successors are Iterable[CpgNode]
+successors: Iterable[CpgNode] = graph.succ(node)
+for succ in successors:
+    succ.code  # Type-safe: IDE knows succ is CpgNode
+```
+
+This ensures that:
+- Return types are preserved throughout graph operations
+- IDE autocomplete works correctly
+- Type checkers (mypy, pyright) can verify type correctness
+
+For more details, see [Generics Documentation](docs/GENERICS.md).
+
+---
+
+## **📚 Usage**
+
+### **1️⃣ Load Graph from Joern CSVs**
+
+```python
+from cpg2py import cpg_graph
+
+# Load graph from CSV files
+graph = cpg_graph("nodes.csv", "rels.csv")
+```
+
+The `cpg_graph` function returns a `CpgGraph` instance, which is the concrete implementation of the graph querier.
+
+---
+
+### **2️⃣ Query Nodes & Edges**
+
+```python
+from cpg2py import CpgGraph, CpgNode, CpgEdge
+
+# Get a specific node (returns CpgNode)
+node: CpgNode = graph.node("2")
+print(node.name, node.type)  # Example output: "/tmp/example.php" AST_TOPLEVEL
+
+# Get a specific edge (returns CpgEdge)
+edge: CpgEdge = graph.edge("2", "3", "ENTRY")
+print(edge.type)  # Output: ENTRY
+```
+
+---
+
+### **3️⃣ Get Node Connections**
+
+```python
+# Get all outgoing edges from a node
+outgoing_edges = graph.succ(node)
+for out_node in outgoing_edges:
+    print(out_node.id, out_node.name)  # out_node is CpgNode
+
+# Get all incoming edges to a node
+incoming_edges = graph.prev(node)
+for in_node in incoming_edges:
+    print(in_node.id, in_node.name)  # in_node is CpgNode
+```
+
+---
+
+### **4️⃣ AST and Flow Queries**
+
+```python
+# Get top-level file node for a given node
+top_file: CpgNode = graph.topfile_node("5")
+print(top_file.name)  # Output: "example.php"
+
+# Get child nodes in the AST hierarchy
+children = graph.children(node)
+print([child.id for child in children])  # children are CpgNode instances
+
+# Get data flow successors
+flow_successors = graph.flow_to(node)
+print([succ.id for succ in flow_successors])  # successors are CpgNode instances
+```
+
+---
+
+## **🛠 Abstract Base Classes (ABC)**
+
+The following abstract base classes (`ABC`) provide interfaces for extending **node**, **edge**, and **graph** querying behavior. All ABCs are imported directly from the main `cpg2py` package.
+
+---
+
+### **🔹 AbcNodeQuerier (Abstract Node Interface)**
+
+This class defines how nodes interact with the graph storage.
+
+```python
+from cpg2py import AbcNodeQuerier, Storage
+
+class MyNodeQuerier(AbcNodeQuerier):
+    def __init__(self, graph: Storage, nid: str):
+        super().__init__(graph, nid)
+
+    @property
+    def name(self):
+        return self.get_property("name")
+```
+
+---
+
+### **🔹 AbcEdgeQuerier (Abstract Edge Interface)**
+
+Defines the querying mechanisms for edges in the graph.
+
+```python
+from cpg2py import AbcEdgeQuerier, Storage
+
+class MyEdgeQuerier(AbcEdgeQuerier):
+    def __init__(self, graph: Storage, f_nid: str, t_nid: str, e_type: str):
+        super().__init__(graph, f_nid, t_nid, e_type)
+
+    @property
+    def type(self):
+        return self.get_property("type")
+```
+
+---
+
+### **🔹 AbcGraphQuerier (Abstract Graph Interface)**
+
+This class provides an interface for implementing custom graph query mechanisms. It's a generic class that supports type-safe operations.
+
+```python
+from cpg2py import AbcGraphQuerier, Storage
+from typing import Optional
+
+class MyGraphQuerier(AbcGraphQuerier[MyNodeQuerier, MyEdgeQuerier]):
+    def node(self, nid: str) -> Optional[MyNodeQuerier]:
+        return MyNodeQuerier(self.storage, nid)
+
+    def edge(self, fid: str, tid: str, eid: str) -> Optional[MyEdgeQuerier]:
+        return MyEdgeQuerier(self.storage, fid, tid, eid)
+```
+
+**Note**: `AbcGraphQuerier` is a generic class parameterized by node and edge types, ensuring type safety throughout graph operations. The concrete implementation `CpgGraph` is defined as `AbcGraphQuerier[CpgNode, CpgEdge]`.
+
+---
+
+## **🔍 Querying The Graph**
+
+After implementing the abstract classes, you can perform advanced queries:
+
+```python
+from cpg2py import Storage
+
+storage = Storage()
+graph = MyGraphQuerier(storage)
+
+# Query node properties
+node = graph.node("5")
+print(node.name)  # Example Output: "main"
+
+# Query edge properties
+edge = graph.edge("5", "6", "FLOWS_TO")
+print(edge.type)  # Output: "FLOWS_TO"
+```
+
+### **Using the Built-in CpgGraph**
+
+You can also use the built-in `CpgGraph` implementation directly:
+
+```python
+from typing import Iterable
+from cpg2py import cpg_graph, CpgGraph, CpgNode, CpgEdge
+
+# Load from CSV files
+graph: CpgGraph = cpg_graph("nodes.csv", "rels.csv")
+
+# Type-safe operations
+node: CpgNode = graph.node("5")
+edge: CpgEdge = graph.edge("5", "6", "FLOWS_TO")
+
+# Type-safe iteration
+successors: Iterable[CpgNode] = graph.succ(node)
+for succ in successors:
+    print(succ.code)  # Type checker knows succ is CpgNode
+```
+
+---
+
+## **🐝 API Reference**
+
+For more detailed API documentation, please see our [APIs doc](docs/APIs.md).
+
+### **Main Package Exports**
+
+All public APIs are available directly from the `cpg2py` package:
+
+```python
+from cpg2py import (
+    # Factory function
+    cpg_graph,
+    
+    # Concrete implementations
+    CpgGraph,
+    CpgNode,
+    CpgEdge,
+    
+    # Abstract base classes
+    AbcGraphQuerier,
+    AbcNodeQuerier,
+    AbcEdgeQuerier,
+    Storage,
+    
+    # Exceptions
+    CPGError,
+    NodeNotFoundError,
+    EdgeNotFoundError,
+    TopFileNotFoundError,
+)
+```
+
+### **Graph Functions**
+
+- `cpg_graph(node_csv: Path, edge_csv: Path, verbose: bool = False) -> CpgGraph`: Loads graph from CSV files and returns a `CpgGraph` instance.
+- `graph.node(nid: str) -> Optional[CpgNode]`: Retrieves a node by ID (returns `CpgNode`).
+- `graph.edge(fid: str, tid: str, eid: str) -> Optional[CpgEdge]`: Retrieves an edge (returns `CpgEdge`).
+- `graph.succ(node: CpgNode) -> Iterable[CpgNode]`: Gets successor nodes.
+- `graph.prev(node: CpgNode) -> Iterable[CpgNode]`: Gets predecessor nodes.
+- `graph.children(node: CpgNode) -> Iterable[CpgNode]`: Gets child nodes via PARENT_OF edges.
+- `graph.parent(node: CpgNode) -> Iterable[CpgNode]`: Gets parent nodes via PARENT_OF edges.
+- `graph.flow_to(node: CpgNode) -> Iterable[CpgNode]`: Gets data flow successors.
+- `graph.flow_from(node: CpgNode) -> Iterable[CpgNode]`: Gets data flow predecessors.
+- `graph.topfile_node(nid: str) -> CpgNode`: Finds the top-level file node.
+
+### **Node Properties (CpgNode)**
+
+- `.id`: Node ID (string).
+- `.name`: Node name.
+- `.type`: Node type.
+- `.code`: Source code content.
+- `.label`: Node label.
+- `.line_num`: Source code line number.
+- `.flags`: List of node flags.
+- `.children_num`: Number of children.
+- `.func_id`: Function ID.
+- `.class_name`: Class name.
+- `.namespace`: Namespace.
+- `.end_num`: End line number.
+- `.comment`: Documentation comment.
+
+### **Edge Properties (CpgEdge)**
+
+- `.id`: Edge ID tuple `(from_node, to_node, edge_type)`.
+- `.start`: Edge start position.
+- `.end`: Edge end position.
+- `.type`: Edge type.
+- `.var`: Variable name (if applicable).
+
+---
+
+## **🌟 License**
+
+This project is licensed under the **MIT License**.
+

cpg2py-1.1.0.dist-info/RECORD

@@ -0,0 +1,17 @@
+cpg2py/__init__.py,sha256=T8Nt4ZLMtCSQjqRINYlYZH9iLiANj3onJqGZk08he4A,2888
+cpg2py/_exceptions.py,sha256=AR2zPeds9fNcCA9iFZGAGluMg3_lgsKHR_4WUmyuD_w,1240
+cpg2py/_logger.py,sha256=HjJAhs4ZAB_IMoBMVRe3KPGS5GRygGy0eNncvIqu8ls,1309
+cpg2py/_abc/__init__.py,sha256=XN7RKxMpnZqvjDAUWxXXLGKY1jrPpKBAxUbq-pydwh4,208
+cpg2py/_abc/edge.py,sha256=CqRz4xmDPplH3lyzRnoWjMj-JgFLFoL8YAJ6XLUsPjA,2571
+cpg2py/_abc/graph.py,sha256=LDF2ylbJYQubUV2LboJlkiQaqWtLW3LsyKBkTJOl4Gs,7795
+cpg2py/_abc/node.py,sha256=s-AQ2oOVtw9wXjBODYtaRKAp8tkPgddMLX2_6eJglcg,1724
+cpg2py/_abc/storage.py,sha256=vUng2nBlhBOJL6YrPTBZaAjkpLdcQLCAW9lOhz7hPxw,6623
+cpg2py/_cpg/__init__.py,sha256=BYNrxYo0G3XoIaUN9llEYkZoCstYJgt88OrJZCkQW7E,126
+cpg2py/_cpg/edge.py,sha256=GcGfwWzQJY0P1GkL8BsQ-vVcsHRlbUp-Le81lPLNt-w,973
+cpg2py/_cpg/graph.py,sha256=Aks2VsDolKdT1NdEz9Vn5ABIWBTPDGvEq7PbrehLVuI,5674
+cpg2py/_cpg/node.py,sha256=hP6jfw9miUyuqksnX9Sr5pz7NYaY4ESigZZ0cmobMcM,1988
+cpg2py-1.1.0.dist-info/LICENSE,sha256=vTjbt7iL1hUilI8E87FoQerEDa9nbpeip26iA6bguHI,1066
+cpg2py-1.1.0.dist-info/METADATA,sha256=VaniBLcK3R1Vp52Iyma3L0NonkN35bH7b41p0H5wJfw,12262
+cpg2py-1.1.0.dist-info/WHEEL,sha256=WnJ8fYhv8N4SYVK2lLYNI6N0kVATA7b0piVUNvqIIJE,91
+cpg2py-1.1.0.dist-info/top_level.txt,sha256=xDY8faKh5Rczvsqb5Jt9Sq-Y7EOImh7jh-m1oVTnH5k,7
+cpg2py-1.1.0.dist-info/RECORD,,

{cpg2py-1.0.5.dist-info → cpg2py-1.1.0.dist-info}/WHEEL

@@ -1,5 +1,5 @@
 Wheel-Version: 1.0
-Generator: setuptools (75.8.0)
+Generator: setuptools (75.3.3)
 Root-Is-Purelib: true
 Tag: py3-none-any
 

cpg2py/abc/edge.py

@@ -1,39 +0,0 @@
-from __future__ import annotations
-import abc
-from .storage import Storage
-from typing import Any, Dict, Optional, Tuple
-
-class AbcEdgeQuerier(abc.ABC): 
-    
-    def __init__(self, graph: Storage, f_nid: str, t_nid: str, e_type: int = 0) -> None:
-        self.__graph: Storage = graph
-        self.__edge_id: Tuple[str, str, str] = (str(f_nid), str(t_nid), str(e_type))
-        if not graph.contains_edge(self.__edge_id): 
-            raise Exception('CANNOT FIND THE NODE ID {nid} IN GRAPH')
-        return None
-
-    @property
-    def edge_id(self) -> Tuple[str, str, int]:
-        return self.__edge_id
-    
-    @property
-    def from_nid(self) -> str: 
-        return self.__edge_id[0]
-
-    @property
-    def to_nid(self) -> str:
-        return self.__edge_id[1]
-    
-    @property
-    def edge_type(self) -> str: 
-        return self.__edge_id[0]
-    
-    @property
-    def properties(self) -> Optional[Dict[str, Any]]: 
-        return self.__graph.get_edge_props(self.__edge_id)
-    
-    def get_property(self, *prop_names: str) -> Optional[Any]: 
-        prop_values = (self.__graph.get_edge_prop(self.__edge_id, p_name) for p_name in prop_names)
-        return next((value for value in prop_values if value is not None), None)
-
-pass

cpg2py/abc/graph.py

@@ -1,98 +0,0 @@
-
-from __future__ import annotations
-
-from .edge import AbcEdgeQuerier
-from .node import AbcNodeQuerier
-
-from typing import Callable, Deque, List, Optional, Iterable
-from collections import deque
-from .storage import Storage
-import abc
-
-class AbcGraphQuerier(abc.ABC): 
-
-    __NodeCondition = Callable[[AbcNodeQuerier], bool]
-    __EdgeCondition = Callable[[AbcEdgeQuerier], bool]
-    __always_true = lambda _: True
-
-    __NodesResult = Iterable[AbcNodeQuerier]
-    __EdgesResult = Iterable[AbcEdgeQuerier]
-
-    def __init__(self, target: Storage, maxdepth: int=-1) -> None:
-        self.__graph: Storage = target
-        self.__maxdepth: int = maxdepth
-
-    @property
-    def storage(self) -> Storage: 
-        return self.__graph
-
-    @abc.abstractmethod
-    def node(self, whose_id_is: str) -> Optional[AbcNodeQuerier]: 
-        raise NotImplementedError
-
-    @abc.abstractmethod
-    def edge(self, fid, tid, eid) -> Optional[AbcEdgeQuerier]: 
-        raise NotImplementedError
-
-    def nodes(self, who_satisifies: __NodeCondition=__always_true)-> __NodesResult: 
-        for nid in self.__graph.get_nodes(): 
-            cur_node = self.node(whose_id_is=nid)
-            if cur_node and who_satisifies(cur_node): yield cur_node 
-        pass
-    
-    def first_node(self, who_satisifies: __NodeCondition=__always_true)-> Optional[AbcNodeQuerier]:
-        return next(self.nodes(who_satisifies), None) 
-
-    def edges(self, who_satisifies: __EdgeCondition = __always_true) -> __EdgesResult: 
-        for from_id, to_id, edge_id in self.__graph.get_edges(): 
-            cur_edge = self.edge(from_id, to_id, edge_id)
-            if cur_edge and who_satisifies(cur_edge): yield cur_edge
-        pass 
-
-    def succ(self, of: AbcNodeQuerier, who_satisifies: __EdgeCondition = __always_true) -> __NodesResult: 
-        for src, dst, type in self.__graph.out_edges(of.id):
-            if not who_satisifies(self.edge(src, dst, type)): continue
-            yield self.node(whose_id_is=dst)
-        pass
-
-    def prev(self, of: AbcNodeQuerier, who_satisifies: __EdgeCondition = __always_true) -> __NodesResult: 
-        for src, dst, type in self.__graph.in_edges(of.id):
-            if not who_satisifies(self.edge(src, dst, type)): continue
-            yield self.node(whose_id_is=src)
-        pass
-
-    def __bfs_search(self, root: AbcNodeQuerier, condition: __EdgeCondition, reverse: bool) -> __NodesResult: 
-        '''
-        return the nodes from (any edge relation) src node by the bfs order (src node will not included)
-        '''
-        if root is None: return 
-        visited_nids: List[str] = list()
-        nodes_queue: Deque[AbcNodeQuerier] = deque([root, None])
-        depth = self.__maxdepth
-        while depth != 0 and len(nodes_queue) > 1: 
-            cur_node = nodes_queue.popleft()
-            if cur_node == None: 
-                nodes_queue.append(None)
-                depth -= 1
-            elif cur_node.id not in visited_nids:
-                visited_nids.append(cur_node.id)
-                if not reverse: n_nodes = self.succ(cur_node, condition)
-                else: n_nodes = self.prev(cur_node, condition)
-                nodes_queue.extend(n_nodes)
-                if root.id != cur_node.id: yield cur_node
-        pass
-
-    def descendants(self, src: AbcNodeQuerier, condition: __EdgeCondition = __always_true) -> __NodesResult: 
-        '''
-        return the result of descendants from (any edge relation) src node by the bfs order (src node will not included)
-        '''
-        return self.__bfs_search(src, condition, reverse=False)
-
-    def ancestors(self, src: AbcNodeQuerier, condition: __EdgeCondition = __always_true) -> __NodesResult:
-        '''
-        return the result of ancestors from (any edge relation) src node by the bfs order (src node will not included)
-        '''
-        return self.__bfs_search(src, condition, reverse=True)
-
-pass
-    

cpg2py/abc/node.py

@@ -1,26 +0,0 @@
-from typing import Any, Dict, Optional
-from .storage import Storage
-import abc
-
-class AbcNodeQuerier(abc.ABC):  
-    
-    def __init__(self, graph: Storage, nid: str) -> None:
-        self.__nid: str = str(nid)
-        self.__graph: Storage = graph
-        if not graph.contains_node(self.__nid): 
-            raise Exception(f'CANNOT FIND THE NODE ID {nid} IN GRAPH')
-        return None
-    
-    @property
-    def node_id(self) -> str:
-        return self.__nid
-
-    @property
-    def properties(self) -> Optional[Dict[str, Any]]: 
-        return self.__graph.get_node_props(self.__nid)
-    
-    def get_property(self, *prop_names: str) -> Optional[Any]: 
-        prop_values = (self.__graph.get_node_prop(self.__nid, p_name) for p_name in prop_names)
-        return next((value for value in prop_values if value is not None), None)
-    
-pass