deepvregulome 0.1.0__tar.gz

deepvregulome-0.1.0/PKG-INFO

@@ -0,0 +1,139 @@
+Metadata-Version: 2.4
+Name: deepvregulome
+Version: 0.1.0
+Summary: DNABERT-based framework for predicting the functional impact of regulatory variants
+Author: Ramana V. Davuluri
+Author-email: Pratik Dutta <pratik.dutta@stonybrook.edu>
+License: CC-BY-NC-4.0
+Project-URL: Homepage, https://github.com/DavuluriLab/DeepVRegulome
+Project-URL: Paper, https://arxiv.org/abs/2511.09026
+Project-URL: Models, https://huggingface.co/duttaprat/DeepVRegulome
+Project-URL: WebApp, https://deepvregulome.streamlit.app
+Keywords: genomics,variant-effect-prediction,dnabert,regulome,transcription-factors,deep-learning
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: torch>=1.10
+Requires-Dist: transformers>=4.20
+Requires-Dist: huggingface-hub>=0.14
+Requires-Dist: pandas>=1.3
+Requires-Dist: numpy>=1.21
+Provides-Extra: genome
+Requires-Dist: pysam>=0.20; extra == "genome"
+Provides-Extra: viz
+Requires-Dist: matplotlib>=3.5; extra == "viz"
+Requires-Dist: seaborn>=0.12; extra == "viz"
+Provides-Extra: vcf
+Requires-Dist: cyvcf2>=0.30; extra == "vcf"
+Provides-Extra: all
+Requires-Dist: pysam>=0.20; extra == "all"
+Requires-Dist: matplotlib>=3.5; extra == "all"
+Requires-Dist: seaborn>=0.12; extra == "all"
+Requires-Dist: cyvcf2>=0.30; extra == "all"
+
+# DeepVRegulome
+![DeepVRegulome Pipeline](assets/flowchart.png)
+
+DeepVRegulome is an end‑to‑end framework for predicting the functional impact of small somatic variants in non‑coding regulatory regions (splice sites and transcription‑factor‑binding sites) using fine‑tuned DNABERT models.
+
+---
+
+## ✨ Key Features
+
+- ✅ DNABERT-based classifiers for:
+  - Splice sites (acceptor, donor)
+  - ~700 TFBS models
+- ✅ Region-aware scoring of somatic variants using Δp and log₂ odds
+- ✅ Batch processing with multiprocessing and BED/VCF support
+- ✅ Interactive Streamlit dashboard with:
+  - Variant tables, plots, and survival analysis
+  - Attention score visualizations
+
+---
+
+📁 Repository Structure
+```
+DeepVRegulome/
+├── .devcontainer/
+├── .streamlit/
+├── data/
+│   └── Brain/
+├── figures/                         # Exported visualizations (e.g. attention maps)
+│   └── attention/
+│       ├── CTCFL/
+│       └── ZNF384/
+├── notebooks/                      # Jupyter notebooks for key pipeline steps
+│   ├── 01_parse_and_merge_vcfs.ipynb            # Merge and parse VCFs
+│   ├── 02_tfbs_intersection.ipynb               # Intersect VCF with TFBS BEDs
+│   ├── 03_dnabert_input_generation.ipynb        # Generate sequences for DNABERT
+│   ├── 04_scoring_candidate_variants.ipynb      # Compute Δp / logOR & rank variants
+│   └── 05_tfbs_attention_motif_visualization.ipynb  # Plot attention scores & motifs
+├── scripts/                       # Shell scripts for batch inference
+│   ├── run_prediction_tfbs.sh                 # Predict with TFBS models
+│   └── run_prediction_splice_acceptor.sh      # Predict with acceptor models
+├── src/
+│   └── deepvregulome/             # Core Python modules
+│       ├── __init__.py
+│       ├── dnabert_data_generation.py         # Wild/mutated seq generation
+│       ├── intersect.py                       # BED/VCF overlap engine
+│       ├── vcf_loader.py                      # VCF parsing utilities
+│       └── config.yaml                        # Centralized path config
+├── streamlit_app/
+│   └── app_variant_clinical_dashboard.py      # Live clinical dashboard
+├── LICENSE
+├── README.md
+├── requirements.txt
+└── .gitignore
+
+```
+## 🧪 Installation
+```bash
+git clone https://github.com/DavuluriLab//DeepVRegulome.git
+cd DeepVRegulome
+python3 -m venv venv && source venv/bin/activate
+pip install -r requirements.txt
+```
+
+
+
+## ⚙️ Typical Pipeline Flow
+| Step | Description | Location |
+|------|-------------|----------|
+| 1️⃣ | Parse + merge somatic VCFs | `01_parse_and_merge_vcfs.ipynb` |
+| 2️⃣ | Intersect variants with TFBS BEDs | `02_tfbs_intersection.ipynb` |
+| 3️⃣ | Generate ref/mutated k-mers for DNABERT | `03_dnabert_input_generation.ipynb` |
+| 4️⃣ | Predict with DNABERT models | `scripts/run_prediction_tfbs.sh` |
+| 5️⃣ | Compute Δp, find candidate variants | `04_scoring_candidate_variants.ipynb` |
+| 6️⃣ | Visualize attention scores and motifs | `05_tfbs_attention_motif_visualization.ipynb` |
+| 7️⃣ | Browse results interactively | `streamlit_app/app_variant_clinical_dashboard.py` |
+
+
+## 📊 Example Outputs
+  * Candidate variant count by TFBS
+  * DNABERT attention heatmaps
+  * High-impact motif shifts due to mutations
+  * Kaplan–Meier plots for clinical stratification
+
+See figures/attention/ for examples like CTCFL.
+
+
+## 🌐 Live Demo
+
+An interactive instance of the DeepVRegulome dashboard is hosted here:
+➡️ **https://davuluri-lab-brainved.streamlit.app/**
+The deployed app lets you browse model performance metrics and variant-effect predictions without installing any software locally.
+
+## 🧬 Model Checkpoints
+Full DNABERT fine-tuned weights (acceptor, donor, and 700 TFBS models) will be deposited in Zenodo and made publicly available immediately upon journal acceptance.
+In the meantime, researchers may request access by emailing pratik.dutta@stonybrook.edu and ramana.davuluri@stonybrookmedicine.edu  with a brief statement of intended use.
+
+## Citation
+If you use DeepVRegulome in your research, please cite:
+
+
+
+## 🧬 Model Checkpoints
+MIT. See [LICENSE](LICENSE) for details.

deepvregulome-0.1.0/README.md

@@ -0,0 +1,103 @@
+# DeepVRegulome
+![DeepVRegulome Pipeline](assets/flowchart.png)
+
+DeepVRegulome is an end‑to‑end framework for predicting the functional impact of small somatic variants in non‑coding regulatory regions (splice sites and transcription‑factor‑binding sites) using fine‑tuned DNABERT models.
+
+---
+
+## ✨ Key Features
+
+- ✅ DNABERT-based classifiers for:
+  - Splice sites (acceptor, donor)
+  - ~700 TFBS models
+- ✅ Region-aware scoring of somatic variants using Δp and log₂ odds
+- ✅ Batch processing with multiprocessing and BED/VCF support
+- ✅ Interactive Streamlit dashboard with:
+  - Variant tables, plots, and survival analysis
+  - Attention score visualizations
+
+---
+
+📁 Repository Structure
+```
+DeepVRegulome/
+├── .devcontainer/
+├── .streamlit/
+├── data/
+│   └── Brain/
+├── figures/                         # Exported visualizations (e.g. attention maps)
+│   └── attention/
+│       ├── CTCFL/
+│       └── ZNF384/
+├── notebooks/                      # Jupyter notebooks for key pipeline steps
+│   ├── 01_parse_and_merge_vcfs.ipynb            # Merge and parse VCFs
+│   ├── 02_tfbs_intersection.ipynb               # Intersect VCF with TFBS BEDs
+│   ├── 03_dnabert_input_generation.ipynb        # Generate sequences for DNABERT
+│   ├── 04_scoring_candidate_variants.ipynb      # Compute Δp / logOR & rank variants
+│   └── 05_tfbs_attention_motif_visualization.ipynb  # Plot attention scores & motifs
+├── scripts/                       # Shell scripts for batch inference
+│   ├── run_prediction_tfbs.sh                 # Predict with TFBS models
+│   └── run_prediction_splice_acceptor.sh      # Predict with acceptor models
+├── src/
+│   └── deepvregulome/             # Core Python modules
+│       ├── __init__.py
+│       ├── dnabert_data_generation.py         # Wild/mutated seq generation
+│       ├── intersect.py                       # BED/VCF overlap engine
+│       ├── vcf_loader.py                      # VCF parsing utilities
+│       └── config.yaml                        # Centralized path config
+├── streamlit_app/
+│   └── app_variant_clinical_dashboard.py      # Live clinical dashboard
+├── LICENSE
+├── README.md
+├── requirements.txt
+└── .gitignore
+
+```
+## 🧪 Installation
+```bash
+git clone https://github.com/DavuluriLab//DeepVRegulome.git
+cd DeepVRegulome
+python3 -m venv venv && source venv/bin/activate
+pip install -r requirements.txt
+```
+
+
+
+## ⚙️ Typical Pipeline Flow
+| Step | Description | Location |
+|------|-------------|----------|
+| 1️⃣ | Parse + merge somatic VCFs | `01_parse_and_merge_vcfs.ipynb` |
+| 2️⃣ | Intersect variants with TFBS BEDs | `02_tfbs_intersection.ipynb` |
+| 3️⃣ | Generate ref/mutated k-mers for DNABERT | `03_dnabert_input_generation.ipynb` |
+| 4️⃣ | Predict with DNABERT models | `scripts/run_prediction_tfbs.sh` |
+| 5️⃣ | Compute Δp, find candidate variants | `04_scoring_candidate_variants.ipynb` |
+| 6️⃣ | Visualize attention scores and motifs | `05_tfbs_attention_motif_visualization.ipynb` |
+| 7️⃣ | Browse results interactively | `streamlit_app/app_variant_clinical_dashboard.py` |
+
+
+## 📊 Example Outputs
+  * Candidate variant count by TFBS
+  * DNABERT attention heatmaps
+  * High-impact motif shifts due to mutations
+  * Kaplan–Meier plots for clinical stratification
+
+See figures/attention/ for examples like CTCFL.
+
+
+## 🌐 Live Demo
+
+An interactive instance of the DeepVRegulome dashboard is hosted here:
+➡️ **https://davuluri-lab-brainved.streamlit.app/**
+The deployed app lets you browse model performance metrics and variant-effect predictions without installing any software locally.
+
+## 🧬 Model Checkpoints
+Full DNABERT fine-tuned weights (acceptor, donor, and 700 TFBS models) will be deposited in Zenodo and made publicly available immediately upon journal acceptance.
+In the meantime, researchers may request access by emailing pratik.dutta@stonybrook.edu and ramana.davuluri@stonybrookmedicine.edu  with a brief statement of intended use.
+
+## Citation
+If you use DeepVRegulome in your research, please cite:
+
+
+
+## 🧬 Model Checkpoints
+MIT. See [LICENSE](LICENSE) for details.

deepvregulome-0.1.0/pyproject.toml

@@ -0,0 +1,51 @@
+[build-system]
+requires = ["setuptools>=64", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "deepvregulome"
+version = "0.1.0"
+description = "DNABERT-based framework for predicting the functional impact of regulatory variants"
+readme = "README.md"
+license = {text = "CC-BY-NC-4.0"}
+requires-python = ">=3.8"
+authors = [
+    {name = "Pratik Dutta", email = "pratik.dutta@stonybrook.edu"},
+    {name = "Ramana V. Davuluri"},
+]
+keywords = [
+    "genomics", "variant-effect-prediction", "dnabert",
+    "regulome", "transcription-factors", "deep-learning",
+]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Science/Research",
+    "Topic :: Scientific/Engineering :: Bio-Informatics",
+    "Programming Language :: Python :: 3",
+]
+
+dependencies = [
+    "torch>=1.10",
+    "transformers>=4.20",
+    "huggingface-hub>=0.14",
+    "pandas>=1.3",
+    "numpy>=1.21",
+]
+
+[project.optional-dependencies]
+genome = ["pysam>=0.20"]
+viz = ["matplotlib>=3.5", "seaborn>=0.12"]
+vcf = ["cyvcf2>=0.30"]
+all = ["pysam>=0.20", "matplotlib>=3.5", "seaborn>=0.12", "cyvcf2>=0.30"]
+
+[project.urls]
+Homepage = "https://github.com/DavuluriLab/DeepVRegulome"
+Paper = "https://arxiv.org/abs/2511.09026"
+Models = "https://huggingface.co/duttaprat/DeepVRegulome"
+WebApp = "https://deepvregulome.streamlit.app"
+
+[project.scripts]
+deepvregulome = "deepvregulome.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]

deepvregulome-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

deepvregulome-0.1.0/src/deepvregulome/__init__.py

@@ -0,0 +1,32 @@
+"""
+DeepVRegulome: DNABERT-based regulatory variant effect prediction.
+
+Usage:
+    from deepvregulome import DVR
+
+    dvr = DVR(genome="/path/to/hg38.fa")
+
+    # Score a single variant
+    result = dvr.score_variant("chr1", 3456782, "A", "TA", models=["CTCF", "SP1"])
+
+    # Score from sequences directly
+    result = dvr.score_sequence(ref_seq, alt_seq, models=["CTCF"])
+
+    # Score a VCF file
+    results = dvr.score_vcf("variants.vcf", models=["CTCF", "SP1", "MYC"])
+"""
+
+__version__ = "0.1.0"
+
+from deepvregulome.registry import ModelRegistry
+
+
+def __getattr__(name):
+    """Lazy import DVR so registry works even without torch installed."""
+    if name == "DVR":
+        from deepvregulome.dvr import DVR
+        return DVR
+    raise AttributeError(f"module 'deepvregulome' has no attribute {name}")
+
+
+__all__ = ["DVR", "ModelRegistry", "__version__"]

deepvregulome-0.1.0/src/deepvregulome/cli.py

@@ -0,0 +1,169 @@
+"""
+Command-line interface for DeepVRegulome.
+
+Usage:
+    # Score a single variant
+    deepvregulome score --chrom chr1 --pos 3456782 --ref A --alt TA \
+        --models CTCFL SP1 MYC --genome hg38.fa
+
+    # Score a VCF file
+    deepvregulome score-vcf variants.vcf --models CTCFL SP1 --genome hg38.fa -o results.tsv
+
+    # Score from sequences
+    deepvregulome score-seq --ref ATCG... --alt ATCG... --models CTCFL
+
+    # List available models
+    deepvregulome list
+    deepvregulome list --type TF
+    deepvregulome search ZNF
+"""
+
+import argparse
+import sys
+
+
+def cmd_score(args):
+    from deepvregulome import DVR
+
+    dvr = DVR(genome=args.genome, device=args.device)
+    result = dvr.score_variant(
+        chrom=args.chrom,
+        pos=args.pos,
+        ref=args.ref,
+        alt=args.alt,
+        models=args.models,
+        model_type=args.type,
+        return_attention=args.attention,
+    )
+
+    if args.output:
+        result.to_csv(args.output, sep="\t", index=False)
+        print(f"Results saved to {args.output}")
+    else:
+        print(result.to_string(index=False))
+
+
+def cmd_score_seq(args):
+    from deepvregulome import DVR
+
+    dvr = DVR(device=args.device)
+    result = dvr.score_sequence(
+        ref_seq=args.ref,
+        alt_seq=args.alt,
+        models=args.models,
+        model_type=args.type,
+        return_attention=args.attention,
+    )
+
+    if args.output:
+        result.to_csv(args.output, sep="\t", index=False)
+    else:
+        print(result.to_string(index=False))
+
+
+def cmd_score_vcf(args):
+    from deepvregulome import DVR
+
+    dvr = DVR(genome=args.genome, device=args.device)
+    result = dvr.score_vcf(
+        vcf_path=args.vcf,
+        models=args.models,
+        model_type=args.type,
+        return_attention=args.attention,
+        max_variants=args.max_variants,
+    )
+
+    output = args.output or args.vcf.replace(".vcf", "_dvr_scores.tsv")
+    result.to_csv(output, sep="\t", index=False)
+    print(f"Scored {len(result)} variant×model combinations → {output}")
+    disrupted = result["disrupted"].sum() if "disrupted" in result.columns else 0
+    print(f"Disrupted: {disrupted}/{len(result)}")
+
+
+def cmd_list(args):
+    from deepvregulome import ModelRegistry
+
+    reg = ModelRegistry()
+    models = reg.list(model_type=args.type)
+    print(f"{'Name':<15} {'Type':<8} {'Acc':>6} {'ROC-AUC':>8} {'Peaks':>8}")
+    print("-" * 50)
+    for m in models:
+        print(f"{m.name:<15} {m.model_type:<8} {m.accuracy:>6.1f} {m.roc_auc:>8.2f} {m.peak_count:>8}")
+    print(f"\nTotal: {len(models)} models")
+
+
+def cmd_search(args):
+    from deepvregulome import ModelRegistry
+
+    reg = ModelRegistry()
+    matches = reg.search(args.query)
+    if matches:
+        print(f"Models matching '{args.query}': {', '.join(matches)}")
+    else:
+        print(f"No models found matching '{args.query}'")
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        prog="deepvregulome",
+        description="DeepVRegulome: Regulatory variant effect prediction",
+    )
+    sub = parser.add_subparsers(dest="command")
+
+    # --- score ---
+    p_score = sub.add_parser("score", help="Score a single variant")
+    p_score.add_argument("--chrom", required=True, help="Chromosome (e.g., chr1)")
+    p_score.add_argument("--pos", type=int, required=True, help="1-based position")
+    p_score.add_argument("--ref", required=True, help="Reference allele")
+    p_score.add_argument("--alt", required=True, help="Alternate allele")
+    p_score.add_argument("--models", nargs="+", help="Model names (e.g., CTCFL SP1)")
+    p_score.add_argument("--type", choices=["TF", "HISTONE"], help="Score all models of this type")
+    p_score.add_argument("--genome", required=True, help="Reference genome FASTA")
+    p_score.add_argument("--attention", action="store_true", help="Include attention scores")
+    p_score.add_argument("--device", default=None, help="cuda or cpu")
+    p_score.add_argument("-o", "--output", help="Output TSV file")
+    p_score.set_defaults(func=cmd_score)
+
+    # --- score-seq ---
+    p_seq = sub.add_parser("score-seq", help="Score from pre-extracted sequences")
+    p_seq.add_argument("--ref", required=True, help="Reference sequence (301bp)")
+    p_seq.add_argument("--alt", required=True, help="Alternate sequence")
+    p_seq.add_argument("--models", nargs="+", help="Model names")
+    p_seq.add_argument("--type", choices=["TF", "HISTONE"], help="Score all of type")
+    p_seq.add_argument("--attention", action="store_true")
+    p_seq.add_argument("--device", default=None)
+    p_seq.add_argument("-o", "--output", help="Output TSV file")
+    p_seq.set_defaults(func=cmd_score_seq)
+
+    # --- score-vcf ---
+    p_vcf = sub.add_parser("score-vcf", help="Score variants from a VCF file")
+    p_vcf.add_argument("vcf", help="Input VCF file")
+    p_vcf.add_argument("--models", nargs="+", help="Model names")
+    p_vcf.add_argument("--type", choices=["TF", "HISTONE"], help="Score all of type")
+    p_vcf.add_argument("--genome", required=True, help="Reference genome FASTA")
+    p_vcf.add_argument("--attention", action="store_true")
+    p_vcf.add_argument("--max-variants", type=int, help="Max variants to score")
+    p_vcf.add_argument("--device", default=None)
+    p_vcf.add_argument("-o", "--output", help="Output TSV file")
+    p_vcf.set_defaults(func=cmd_score_vcf)
+
+    # --- list ---
+    p_list = sub.add_parser("list", help="List available models")
+    p_list.add_argument("--type", choices=["TF", "HISTONE"], help="Filter by type")
+    p_list.set_defaults(func=cmd_list)
+
+    # --- search ---
+    p_search = sub.add_parser("search", help="Search model names")
+    p_search.add_argument("query", help="Search string (e.g., ZNF, GATA)")
+    p_search.set_defaults(func=cmd_search)
+
+    args = parser.parse_args()
+    if not args.command:
+        parser.print_help()
+        sys.exit(1)
+
+    args.func(args)
+
+
+if __name__ == "__main__":
+    main()

deepvregulome-0.1.0/src/deepvregulome/dnabert_data_generation.py

@@ -0,0 +1,151 @@
+import os, glob
+import pandas as pd
+import pysam
+import numpy as np
+import pickle
+from multiprocessing import Pool, cpu_count
+
+# === Configurable paths ===
+cancer_type = "Brain"
+intersected_base_path = f"/data/private/pdutta_new/GDC_Cancer_Wise/New_data/{cancer_type}/Generated_files/Intersected_Data/Somatic/300bp_TFBS"
+reference_genome_path = "/data/projects/Resources/Gencode_genome_annotation/GRCh38.primary_assembly.genome.fa"
+output_path = f"/data/private/pdutta_new/GDC_Cancer_Wise/New_data/{cancer_type}/Generated_files/DNABERT_Data/Somatic/TFBS_300bp_new"
+
+os.makedirs(output_path, exist_ok=True)
+
+# === Load top models list ===
+top_models_df = pd.read_csv("/home/campus.stonybrook.edu/pdutta/Github/Postdoc/DNABERT_data_processing/TFBS/300bp_TFBS_accuracy_Stat.tsv", sep="\t")
+top_models_df = top_models_df[top_models_df['eval_acc'] >= 0.85].iloc[100:200]
+
+# === Convert sequence to space-separated k-mers ===
+def seq2kmer(seq, k=6):
+    return " ".join([seq[i:i+k] for i in range(len(seq)-k+1)])
+
+# === Apply all mutations to a given reference sequence ===
+def apply_mutations_absolute(ref_seq, chip_start, starts, ends, refs, alts):
+    mutations = sorted(zip(starts, ends, refs, alts), key=lambda x: x[0])
+    seq_list = list(ref_seq)
+    offset = chip_start
+    for start, end, ref, alt in mutations:
+        rel_pos = start - offset
+        if ''.join(seq_list[rel_pos:rel_pos+len(ref)]) == ref:
+            seq_list[rel_pos:rel_pos+len(ref)] = list(alt)
+    return ''.join(seq_list)
+
+# === Load genome reference once per worker ===
+def initialize_worker():
+    global reference_fasta
+    reference_fasta = pysam.FastaFile(reference_genome_path)
+    print("##### Reference genome loaded in worker #####")
+
+# === Process a dataframe of variants for a single TFBS-patient combo ===
+def get_sequences(df):
+    global reference_fasta
+    try:
+        data = []
+        for idx, row in df.iterrows():
+            chrom, ref_start, ref_end = row[0], row[1], row[2]
+            variant_start, variant_end = row['START_POS'], row['END_POS']
+            ref_nucleotide, alt = row['REF'], row['ALT']
+            ref_seq = reference_fasta.fetch(chrom, ref_start, ref_end)
+            variant_pos_start = variant_start - ref_start
+            variant_pos_end = variant_end - ref_start
+
+            if len(ref_nucleotide) < len(alt):  # Insertion
+                delete_size = len(alt) - len(ref_nucleotide)
+                alt_seq = ref_seq[:variant_pos_start] + alt + ref_seq[variant_pos_end:len(ref_seq)-delete_size]
+            elif len(ref_nucleotide) > len(alt):  # Deletion
+                insert_size = len(ref_nucleotide) - len(alt)
+                extra_bases = reference_fasta.fetch(chrom, ref_end, ref_end+insert_size)
+                alt_seq = ref_seq[:variant_pos_start] + alt + ref_seq[variant_pos_end:] + extra_bases
+            else:  # SNV
+                alt_seq = ref_seq[:variant_pos_start] + alt + ref_seq[variant_pos_end:]
+
+            data.append({
+                'chr': chrom, 'Chip_Seq_start': ref_start, 'Chip_Seq_end': ref_end,
+                'varinat_start': variant_start, 'variant_end': variant_end,
+                'ref_neucleotide': ref_nucleotide, 'alternative_neucleotide': alt,
+                'reference_seq': ref_seq, 'alt_seq': alt_seq
+            })
+
+        new_df = pd.DataFrame(data).drop_duplicates().reset_index(drop=True)
+
+        merged_list = list(zip(new_df['reference_seq'], new_df['alt_seq']))
+        merged_list = [item.upper() for tup in merged_list for item in tup]
+        df_kmer = pd.DataFrame(list(map(seq2kmer, merged_list)), columns=['Sequence'])
+        df_kmer['Label'] = np.random.choice([0, 1], size=len(df_kmer))
+
+        grouped_df = new_df.groupby(['chr', 'Chip_Seq_start', 'Chip_Seq_end']).agg({
+            'varinat_start': list, 'variant_end': list,
+            'ref_neucleotide': list, 'alternative_neucleotide': list,
+            'reference_seq': 'first', 'alt_seq': lambda x: list(set(x))
+        }).reset_index()
+
+        grouped_df['mutated_sequence'] = grouped_df.apply(lambda row: apply_mutations_absolute(
+            row['reference_seq'], row['Chip_Seq_start'], row['varinat_start'],
+            row['variant_end'], row['ref_neucleotide'], row['alternative_neucleotide']), axis=1)
+
+        merged_regionwise = list(zip(grouped_df['reference_seq'], grouped_df['mutated_sequence']))
+        merged_regionwise = [item.upper() for tup in merged_regionwise for item in tup]
+        df_kmer_region = pd.DataFrame(list(map(seq2kmer, merged_regionwise)), columns=['Sequence'])
+        df_kmer_region['Label'] = np.random.choice([0, 1], size=len(df_kmer_region))
+
+        return new_df, df_kmer, grouped_df, df_kmer_region
+
+    except Exception as e:
+        print(f"Error in get_sequences: {e}")
+        raise
+
+# === Main processing logic for each TFBS tag ===
+def process_row(row):
+    global reference_fasta
+    try:
+        print(f"Processing {row['tags']}")
+        intersected_data = f"{intersected_base_path}/{row['tags']}/intersected_vcf_data.pkl"
+
+        with open(intersected_data, "rb") as file:
+            loaded_dictionary = pickle.load(file)
+
+        dnabert_raw_data = {}
+        dnabert_raw_data_regionwise = {}
+
+        for key, value in loaded_dictionary.items():
+            try:
+                print(f"  → {key}")
+                new_df, df_kmer, grouped_df, df_kmer_region = get_sequences(value)
+                dnabert_raw_data[key] = new_df
+                dnabert_raw_data_regionwise[key] = grouped_df
+
+                patient_folder = f"{output_path}/{row['tags']}/Patient_wise"
+                os.makedirs(f"{patient_folder}/variant_wise/{key}", exist_ok=True)
+                os.makedirs(f"{patient_folder}/region_wise/{key}", exist_ok=True)
+                df_kmer.to_csv(f"{patient_folder}/variant_wise/{key}/dev.tsv", sep="\t", index=False)
+                df_kmer_region.to_csv(f"{patient_folder}/region_wise/{key}/dev.tsv", sep="\t", index=False)
+
+            except Exception as e:
+                print(f"Error: {e} in {row['tags']} → {key}")
+                return row['tags']
+
+        with open(f"{output_path}/{row['tags']}/variantwise_raw_vcf_data.pkl", "wb") as file:
+            pickle.dump(dnabert_raw_data, file)
+        with open(f"{output_path}/{row['tags']}/regionwise_raw_vcf_data.pkl", "wb") as file:
+            pickle.dump(dnabert_raw_data_regionwise, file)
+
+    except Exception as e:
+        print(f"Global error in {row['tags']}: {e}")
+        return row['tags']
+
+    return None
+
+# === Run multiprocessing ===
+if __name__ == '__main__':
+    print(f"Using {cpu_count()} CPUs for processing")
+    with Pool(cpu_count() - 10, initializer=initialize_worker) as pool:
+        missing_files = pool.map(process_row, [row for _, row in top_models_df.iterrows()])
+
+    missing_files = [tag for tag in missing_files if tag is not None]
+    with open(f"{output_path}/missing_files.txt", "w") as f:
+        for tag in missing_files:
+            f.write(f"{tag}\n")
+
+    print(f"Missing files recorded at {output_path}/missing_files.txt")