gnn-vuln 0.1.1__tar.gz

gnn_vuln-0.1.1/.gitignore

@@ -0,0 +1,104 @@
+# Python
+__pycache__/
+*.py[cod]
+*.pyo
+*.pyd
+*.so
+*.egg
+*.egg-info/
+dist/
+build/
+.eggs/
+
+# uv / virtualenv
+.venv/
+.uv/
+
+# embedded Joern clone (nested .git, large) + local materialize cache
+src/gnn_vuln/joern/
+data/.materialized/
+
+# Jupyter
+.ipynb_checkpoints/
+*.ipynb_metadata/
+
+
+# Logs
+logs/
+*.log
+
+# OS
+.DS_Store
+Thumbs.db
+
+# IDE
+.vscode/settings.json
+.idea/
+
+# Claude Code local config (commands, hooks — project-local, not for repo)
+.claude/
+.serena/
+.agents/
+
+# Env files
+.env
+*.env
+
+# Joern output cache
+joern-cli/
+joern_workspace/
+
+# Large generated / binary / third-party — do NOT push
+/data/raw/**
+!/data/raw/*/
+!/data/raw/*/cwe_vocab.json
+/data/datasets/
+/data/processed/
+/data/graphs/
+/checkpoints/
+/results/
+/paper/
+# Baselines tracked in-repo (LineVul, LineVD, LOSVER) so the pod gets them via git pull.
+# Ignore only their generated junk:
+src/*/__pycache__/
+src/**/*.pyc
+src/LineVul/saved_models/
+src/linevd/storage/cache/
+src/linevd/storage/processed/
+src/VulChecker/
+src/vul-LMGNN/
+src/gnn_vuln/joern/
+src/SCLCVD/
+src/EDAT/
+
+# Downloaded tools / secrets
+gdrive.exe
+gdrive3.zip
+gdrive*.zip
+rclone.zip
+rclone*/
+secret.txt
+*.secret
+
+.kiro/
+src/GNNPlus/
+src/Graph-Mixture-of-Experts/
+src/mixture-of-experts/
+src/CodeGraphSMOTE/
+src/GMixup/
+src/GraphSMOTE/
+src/FLAG/
+src/graph-jepa/
+src/graph-vit-mlpmixer/
+src/llm-jepa/
+src/ijepa/
+src/GraphMAE/
+src/GraphMAE2/
+src/bgrl/
+src/StructMAE/
+src/hypermixing/
+src/pnlp-mixer/
+src/GraphGPS/
+src/LineVul/cppcheck/
+/data/baselines/
+docs/

gnn_vuln-0.1.1/PKG-INFO

@@ -0,0 +1,204 @@
+Metadata-Version: 2.4
+Name: gnn-vuln
+Version: 0.1.1
+Summary: GNN-based vulnerability detection for code — Final Project (Tugas Akhir)
+Author: Otzzu
+License: MIT
+Requires-Python: >=3.11
+Requires-Dist: datasets>=4.8.4
+Requires-Dist: h5py>=3.16.0
+Requires-Dist: ijson>=3.5.0
+Requires-Dist: loguru>=0.7.2
+Requires-Dist: matplotlib>=3.8.0
+Requires-Dist: networkx>=3.3
+Requires-Dist: numpy>=1.26.0
+Requires-Dist: pandas>=2.2.0
+Requires-Dist: pdfplumber>=0.11.9
+Requires-Dist: python-docx>=1.2.0
+Requires-Dist: pyyaml>=6.0.1
+Requires-Dist: scikit-learn>=1.4.0
+Requires-Dist: seaborn>=0.13.0
+Requires-Dist: torch-geometric>=2.5.0
+Requires-Dist: torch>=2.2.0
+Requires-Dist: tqdm>=4.66.0
+Requires-Dist: transformers<5.0,>=4.48
+Requires-Dist: tree-sitter-c>=0.23.0
+Requires-Dist: tree-sitter>=0.23.0
+Provides-Extra: dev
+Requires-Dist: ipykernel>=6.29.0; extra == 'dev'
+Requires-Dist: jupyter>=1.0.0; extra == 'dev'
+Requires-Dist: jupyterlab>=4.0.0; extra == 'dev'
+Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
+Requires-Dist: pytest>=8.0.0; extra == 'dev'
+Requires-Dist: ruff>=0.4.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# `gnn_vuln` — Library API Reference
+
+The installable model library behind the vulnerability-detection service. This is the
+complete public surface: what to import, the inputs, and the outputs.
+
+**Not everything is file-based.** You pass a function **source string** and get a **result
+dict** back. The only files involved are the model checkpoint + config (normal — weights and
+config live on disk) and the Joern CPG, which is created in a private temp dir and hidden
+from you. In-memory in, in-memory out.
+
+---
+
+## Install
+
+```bash
+# 1. torch + PyG sparse ext from their own indexes (PyPI can't resolve these alone)
+pip install torch==2.6.0 --index-url https://download.pytorch.org/whl/cpu     # or cu124
+pip install torch-scatter torch-sparse -f https://data.pyg.org/whl/torch-2.6.0+cpu.html
+# 2. the library
+pip install gnn-vuln
+```
+
+Plus **Joern** (CPG generation) + a **JDK 21** on the host. Point the predictor at the
+`joern-cli` directory.
+
+---
+
+## Inference — `gnn_vuln.inference`
+
+### `VulnPredictor` (high-level, recommended)
+
+```python
+from gnn_vuln.inference import VulnPredictor
+
+predictor = VulnPredictor.from_checkpoint(
+    checkpoint="checkpoints/<run>/best_model.pt",   # trained weights (.pt file)
+    config="configs/<arch>/config.yaml",            # its config (file, or pass a list)
+    device="cuda",                                  # "cpu" | "cuda"
+)
+predictor.class_names = ["benign", "CWE-787", ...]  # optional: override label names
+```
+
+| Method | Input | Output |
+| --- | --- | --- |
+| `predict_code(code, joern_cli, max_nodes=2500, top_k_lines=None)` | function **source string** | result `dict`, or `None` if Joern produced no CPG |
+| `predict_codes(codes, joern_cli, max_nodes=2500, top_k_lines=None)` | `list[str]` | list of result dicts (`None` per entry on Joern failure) |
+| `predict(data, top_k_lines=None)` | a PyG `Data` object (already built) | result `dict` |
+| `predict_from_file(cpg_path, max_nodes=1000, top_k_lines=None)` | path to a Joern CPG file | result `dict`, or `None` |
+
+```python
+# the everyday call — string in, dict out (Joern handled internally)
+result = predictor.predict_code(
+    "void f(char *s){ char b[8]; strcpy(b, s); }",
+    joern_cli="C:/joern/joern-cli",
+    top_k_lines=5,
+)
+```
+
+### Result dict (schema)
+
+```python
+{
+  "prediction":          "CWE-120",          # predicted class name
+  "class_id":            7,                   # predicted class index
+  "is_vulnerable":       True,                # class_id > 0
+  "confidence":          0.87,                # softmax prob of the predicted class [0,1]
+  "class_probabilities": {"benign": 0.01, "CWE-120": 0.87, ...},
+  "suspicious_lines":    [{"line": 3, "score": 0.92, "code": "strcpy(b, s);"}, ...],  # score-desc
+  "cls_embedding":       [0.013, -0.44, ...], # pre-head function vector (for search/drift)
+}
+```
+
+`suspicious_lines` may also carry `predicted_cwe` + per-line `class_probabilities` for the
+multiclass statement head. `cls_embedding` is the representation fed to the output head.
+
+### Module functions (lower-level)
+
+```python
+from gnn_vuln.inference import load_model, predict, predict_from_file
+
+model, class_names = load_model(checkpoint, config, device="cpu")   # -> (nn.Module, list[str])
+result = predict(model, data, class_names, device=None, top_k_lines=None)   # PyG Data -> dict
+result = predict_from_file(model, cpg_path, class_names, pretrained_lm=..., ...)  # file -> dict
+```
+
+---
+
+## CPG generation — `gnn_vuln.data.joern_runner`
+
+Only needed if you want the CPG file yourself; `predict_code` calls this for you.
+
+```python
+from gnn_vuln.data.joern_runner import process_function
+from pathlib import Path
+
+cpg_path = process_function(
+    code="int add(int a,int b){return a+b;}",  # source string
+    idx=0,
+    out_dir=Path("./out"),
+    joern_cli_dir=Path("C:/joern/joern-cli"),
+    fmt="graphml",         # "graphml" | "json"
+    lang=None,             # None = auto-detect (c/cpp/java/js/py)
+)   # -> Path to the written CPG, or None on failure
+```
+
+---
+
+## Config — `gnn_vuln.config`
+
+```python
+from gnn_vuln.config import Config
+
+cfg = Config.from_yaml("N48.yaml")                              # one monolithic file
+cfg = Config.from_yamls(["data.yaml", "model.yaml", "train.yaml"])  # split, merged in order
+# cfg.data, cfg.model, cfg.train, cfg.ewc, cfg.replay  — dataclasses
+cfg.data.mode          # "binary" | "multiclass"
+cfg.model.architecture # "lmgat_codebert" | "lmgat_seqgnn"
+cfg.train.epochs       # 100
+```
+
+`from_yamls` lets you split data / model / train configs into separate files; a single file
+is just the one-element case (identical behaviour).
+
+---
+
+## Data pipeline & training — module CLIs (`python -m`)
+
+Each step is a runnable module. All accept **one** config file or **several** split files
+(merged section-by-section).
+
+| Command                                                                                                 | In                 | Out                                       |
+| ------------------------------------------------------------------------------------------------------- | ------------------ | ----------------------------------------- |
+| `python -m gnn_vuln.data.prepare --input <parquet> --format bigvul --out-dir <dir> --joern-cli <joern>` | raw rows (parquet) | per-function CPGs + `cwe_vocab.json`      |
+| `python -m gnn_vuln.data.build_pt --config <yaml…> --split train`                                       | CPG dir            | processed `.pt` (UniXcoder node features) |
+| `python -m gnn_vuln.data.merge --config <yaml…> --sources <s1> <s2> … --out-source <name> [--dedup]`     | built `.pt`s       | one merged `.pt` (label space unified)    |
+| `python -m gnn_vuln.train --config <yaml…>`                                                             | `.pt` + config     | trained checkpoint + metrics              |
+
+`prepare` flags: `--binary`, `--top-cwe N`, `--sample-per-class N`, `--workers N`.
+Installed console scripts: `train`, `evaluate` (= `python -m gnn_vuln.train` / `.evaluate`).
+
+The whole raw→pt→train flow:
+
+```bash
+python -m gnn_vuln.data.prepare  --input data.parquet --format bigvul --out-dir data/raw --joern-cli <joern>
+python -m gnn_vuln.data.build_pt --config config.yaml --split train
+python -m gnn_vuln.train         --config config.yaml
+```
+
+---
+
+## Package layout
+
+```
+gnn_vuln/
+  inference.py            VulnPredictor, load_model, predict, predict_from_file
+  config.py               Config (data/model/train/ewc/replay), from_yaml / from_yamls
+  train.py                trainer  (python -m gnn_vuln.train)
+  evaluate.py             evaluation (python -m gnn_vuln.evaluate)
+  models/                 lmgat_codebert, lmgat_seqgnn — the architectures (built via config)
+  data/
+    prepare.py            raw rows → Joern CPG            (python -m)
+    build_pt.py           CPG → .pt                       (python -m)
+    joern_runner.py       process_function — Joern wrapper
+    dataset_lm.py         CodeBERTGraphDataset (PyG InMemoryDataset, UniXcoder features)
+    node_embedder.py      frozen LM per-node embeddings
+```
+
+The library resolves its data/checkpoint root from `$GNN_VULN_ROOT` (else the current working
+directory), so it behaves the same installed-from-PyPI as in a source checkout.

gnn_vuln-0.1.1/pyproject.toml

@@ -0,0 +1,98 @@
+[project]
+name = "gnn-vuln"
+version = "0.1.1"
+description = "GNN-based vulnerability detection for code — Final Project (Tugas Akhir)"
+readme = "src/gnn_vuln/README.md"
+requires-python = ">=3.11"
+license = { text = "MIT" }
+authors = [{ name = "Otzzu" }]
+
+dependencies = [
+    # Deep learning
+    "torch>=2.2.0",
+    # Graph neural networks (PyTorch Geometric)
+    "torch-geometric>=2.5.0",
+    # Graph utilities
+    "networkx>=3.3",
+    # Data science
+    "numpy>=1.26.0",
+    "pandas>=2.2.0",
+    "scikit-learn>=1.4.0",
+    # Visualisation
+    "matplotlib>=3.8.0",
+    "seaborn>=0.13.0",
+    # Progress & config
+    "tqdm>=4.66.0",
+    "pyyaml>=6.0.1",
+    # Parsing (AST extraction helper)
+    "tree-sitter>=0.23.0",
+    "tree-sitter-c>=0.23.0",
+    # Logging
+    "loguru>=0.7.2",
+    "datasets>=4.8.4",
+    "transformers>=4.48,<5.0",
+    "pdfplumber>=0.11.9",
+    "ijson>=3.5.0",
+    "python-docx>=1.2.0",
+    "h5py>=3.16.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "jupyter>=1.0.0",
+    "ipykernel>=6.29.0",
+    "jupyterlab>=4.0.0",
+    "pytest>=8.0.0",
+    "pytest-cov>=5.0.0",
+    "ruff>=0.4.0",
+]
+
+[project.scripts]
+train = "gnn_vuln.train:main"
+evaluate = "gnn_vuln.evaluate:main"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/gnn_vuln"]
+
+# Bound the sdist to the package only. Without this hatchling walks the whole src/ tree
+# (LIVABLE, MVulD, VulPCL, … baseline clones — GBs) into the source tarball, which is
+# slow and wrong. pyproject.toml + README.md are always added by hatchling.
+[tool.hatch.build.targets.sdist]
+only-include = ["src/gnn_vuln"]
+
+# ---------------------------------------------------------------------------
+# Tool configuration
+# ---------------------------------------------------------------------------
+
+[tool.ruff]
+line-length = 100
+target-version = "py311"
+src = ["src"]
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "UP"]
+ignore = ["E501"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-v --tb=short"
+
+# NOTE: PyTorch Geometric sparse extensions (torch-scatter, torch-sparse)
+# require matching torch+cpu wheels. Install them AFTER `uv sync` with:
+#   uv run pip install torch-scatter torch-sparse -f https://data.pyg.org/whl/torch-2.2.0+cpu.html
+# (See README.md Quick Start for full instructions.)
+
+# ---------------------------------------------------------------------------
+# PyTorch CUDA source (RTX 4060 / CUDA 12.4)
+# ---------------------------------------------------------------------------
+[tool.uv.sources]
+torch = { index = "pytorch-cu124" }
+
+[[tool.uv.index]]
+name = "pytorch-cu124"
+url = "https://download.pytorch.org/whl/cu124"
+explicit = true

gnn_vuln-0.1.1/src/gnn_vuln/README.md

@@ -0,0 +1,169 @@
+# `gnn_vuln` — Library API Reference
+
+The installable model library behind the vulnerability-detection service. This is the
+complete public surface: what to import, the inputs, and the outputs.
+
+**Not everything is file-based.** You pass a function **source string** and get a **result
+dict** back. The only files involved are the model checkpoint + config (normal — weights and
+config live on disk) and the Joern CPG, which is created in a private temp dir and hidden
+from you. In-memory in, in-memory out.
+
+---
+
+## Install
+
+```bash
+# 1. torch + PyG sparse ext from their own indexes (PyPI can't resolve these alone)
+pip install torch==2.6.0 --index-url https://download.pytorch.org/whl/cpu     # or cu124
+pip install torch-scatter torch-sparse -f https://data.pyg.org/whl/torch-2.6.0+cpu.html
+# 2. the library
+pip install gnn-vuln
+```
+
+Plus **Joern** (CPG generation) + a **JDK 21** on the host. Point the predictor at the
+`joern-cli` directory.
+
+---
+
+## Inference — `gnn_vuln.inference`
+
+### `VulnPredictor` (high-level, recommended)
+
+```python
+from gnn_vuln.inference import VulnPredictor
+
+predictor = VulnPredictor.from_checkpoint(
+    checkpoint="checkpoints/<run>/best_model.pt",   # trained weights (.pt file)
+    config="configs/<arch>/config.yaml",            # its config (file, or pass a list)
+    device="cuda",                                  # "cpu" | "cuda"
+)
+predictor.class_names = ["benign", "CWE-787", ...]  # optional: override label names
+```
+
+| Method | Input | Output |
+| --- | --- | --- |
+| `predict_code(code, joern_cli, max_nodes=2500, top_k_lines=None)` | function **source string** | result `dict`, or `None` if Joern produced no CPG |
+| `predict_codes(codes, joern_cli, max_nodes=2500, top_k_lines=None)` | `list[str]` | list of result dicts (`None` per entry on Joern failure) |
+| `predict(data, top_k_lines=None)` | a PyG `Data` object (already built) | result `dict` |
+| `predict_from_file(cpg_path, max_nodes=1000, top_k_lines=None)` | path to a Joern CPG file | result `dict`, or `None` |
+
+```python
+# the everyday call — string in, dict out (Joern handled internally)
+result = predictor.predict_code(
+    "void f(char *s){ char b[8]; strcpy(b, s); }",
+    joern_cli="C:/joern/joern-cli",
+    top_k_lines=5,
+)
+```
+
+### Result dict (schema)
+
+```python
+{
+  "prediction":          "CWE-120",          # predicted class name
+  "class_id":            7,                   # predicted class index
+  "is_vulnerable":       True,                # class_id > 0
+  "confidence":          0.87,                # softmax prob of the predicted class [0,1]
+  "class_probabilities": {"benign": 0.01, "CWE-120": 0.87, ...},
+  "suspicious_lines":    [{"line": 3, "score": 0.92, "code": "strcpy(b, s);"}, ...],  # score-desc
+  "cls_embedding":       [0.013, -0.44, ...], # pre-head function vector (for search/drift)
+}
+```
+
+`suspicious_lines` may also carry `predicted_cwe` + per-line `class_probabilities` for the
+multiclass statement head. `cls_embedding` is the representation fed to the output head.
+
+### Module functions (lower-level)
+
+```python
+from gnn_vuln.inference import load_model, predict, predict_from_file
+
+model, class_names = load_model(checkpoint, config, device="cpu")   # -> (nn.Module, list[str])
+result = predict(model, data, class_names, device=None, top_k_lines=None)   # PyG Data -> dict
+result = predict_from_file(model, cpg_path, class_names, pretrained_lm=..., ...)  # file -> dict
+```
+
+---
+
+## CPG generation — `gnn_vuln.data.joern_runner`
+
+Only needed if you want the CPG file yourself; `predict_code` calls this for you.
+
+```python
+from gnn_vuln.data.joern_runner import process_function
+from pathlib import Path
+
+cpg_path = process_function(
+    code="int add(int a,int b){return a+b;}",  # source string
+    idx=0,
+    out_dir=Path("./out"),
+    joern_cli_dir=Path("C:/joern/joern-cli"),
+    fmt="graphml",         # "graphml" | "json"
+    lang=None,             # None = auto-detect (c/cpp/java/js/py)
+)   # -> Path to the written CPG, or None on failure
+```
+
+---
+
+## Config — `gnn_vuln.config`
+
+```python
+from gnn_vuln.config import Config
+
+cfg = Config.from_yaml("N48.yaml")                              # one monolithic file
+cfg = Config.from_yamls(["data.yaml", "model.yaml", "train.yaml"])  # split, merged in order
+# cfg.data, cfg.model, cfg.train, cfg.ewc, cfg.replay  — dataclasses
+cfg.data.mode          # "binary" | "multiclass"
+cfg.model.architecture # "lmgat_codebert" | "lmgat_seqgnn"
+cfg.train.epochs       # 100
+```
+
+`from_yamls` lets you split data / model / train configs into separate files; a single file
+is just the one-element case (identical behaviour).
+
+---
+
+## Data pipeline & training — module CLIs (`python -m`)
+
+Each step is a runnable module. All accept **one** config file or **several** split files
+(merged section-by-section).
+
+| Command                                                                                                 | In                 | Out                                       |
+| ------------------------------------------------------------------------------------------------------- | ------------------ | ----------------------------------------- |
+| `python -m gnn_vuln.data.prepare --input <parquet> --format bigvul --out-dir <dir> --joern-cli <joern>` | raw rows (parquet) | per-function CPGs + `cwe_vocab.json`      |
+| `python -m gnn_vuln.data.build_pt --config <yaml…> --split train`                                       | CPG dir            | processed `.pt` (UniXcoder node features) |
+| `python -m gnn_vuln.data.merge --config <yaml…> --sources <s1> <s2> … --out-source <name> [--dedup]`     | built `.pt`s       | one merged `.pt` (label space unified)    |
+| `python -m gnn_vuln.train --config <yaml…>`                                                             | `.pt` + config     | trained checkpoint + metrics              |
+
+`prepare` flags: `--binary`, `--top-cwe N`, `--sample-per-class N`, `--workers N`.
+Installed console scripts: `train`, `evaluate` (= `python -m gnn_vuln.train` / `.evaluate`).
+
+The whole raw→pt→train flow:
+
+```bash
+python -m gnn_vuln.data.prepare  --input data.parquet --format bigvul --out-dir data/raw --joern-cli <joern>
+python -m gnn_vuln.data.build_pt --config config.yaml --split train
+python -m gnn_vuln.train         --config config.yaml
+```
+
+---
+
+## Package layout
+
+```
+gnn_vuln/
+  inference.py            VulnPredictor, load_model, predict, predict_from_file
+  config.py               Config (data/model/train/ewc/replay), from_yaml / from_yamls
+  train.py                trainer  (python -m gnn_vuln.train)
+  evaluate.py             evaluation (python -m gnn_vuln.evaluate)
+  models/                 lmgat_codebert, lmgat_seqgnn — the architectures (built via config)
+  data/
+    prepare.py            raw rows → Joern CPG            (python -m)
+    build_pt.py           CPG → .pt                       (python -m)
+    joern_runner.py       process_function — Joern wrapper
+    dataset_lm.py         CodeBERTGraphDataset (PyG InMemoryDataset, UniXcoder features)
+    node_embedder.py      frozen LM per-node embeddings
+```
+
+The library resolves its data/checkpoint root from `$GNN_VULN_ROOT` (else the current working
+directory), so it behaves the same installed-from-PyPI as in a source checkout.

gnn_vuln-0.1.1/src/gnn_vuln/__init__.py

@@ -0,0 +1,8 @@
+"""
+gnn_vuln — GNN-based Vulnerability Detection
+=============================================
+Final project package.
+"""
+
+__version__ = "0.1.1"
+__author__ = "Otzzu"