smftools 0.1.7__tar.gz → 0.2.3__tar.gz

{smftools-0.1.7 → smftools-0.2.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: smftools
-Version: 0.1.7
+Version: 0.2.3
 Summary: Single Molecule Footprinting Analysis in Python.
 Project-URL: Source, https://github.com/jkmckenna/smftools
 Project-URL: Documentation, https://smftools.readthedocs.io/
@@ -43,9 +43,11 @@ Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
 Classifier: Topic :: Scientific/Engineering :: Visualization
-Requires-Python: >=3.9
+Requires-Python: <3.13,>=3.9
 Requires-Dist: anndata>=0.10.0
 Requires-Dist: biopython>=1.79
+Requires-Dist: captum
+Requires-Dist: click
 Requires-Dist: fastcluster
 Requires-Dist: hydra-core
 Requires-Dist: igraph
@@ -57,15 +59,18 @@ Requires-Dist: numpy<2,>=1.22.0
 Requires-Dist: omegaconf
 Requires-Dist: pandas>=1.4.2
 Requires-Dist: pod5>=0.1.21
-Requires-Dist: pomegranate>=1.0.0
+Requires-Dist: pybedtools>=0.12.0
+Requires-Dist: pybigwig>=0.3.24
 Requires-Dist: pyfaidx>=0.8.0
 Requires-Dist: pysam>=0.19.1
 Requires-Dist: scanpy>=1.9
 Requires-Dist: scikit-learn>=1.0.2
 Requires-Dist: scipy>=1.7.3
 Requires-Dist: seaborn>=0.11
+Requires-Dist: shap
 Requires-Dist: torch>=1.9.0
 Requires-Dist: tqdm
+Requires-Dist: upsetplot
 Requires-Dist: wandb
 Provides-Extra: docs
 Requires-Dist: ipython>=7.20; extra == 'docs'
@@ -98,12 +103,9 @@ While most genomic data structures handle low-coverage data (<100X) along large
 
 ## Dependencies
 The following CLI tools need to be installed and configured before using the informatics (smftools.inform) module of smftools:
-1) [Dorado](https://github.com/nanoporetech/dorado) -> For standard/modified basecalling and alignment. Can be attained by downloading and configuring nanopore MinKnow software.
-2) [Samtools](https://github.com/samtools/samtools) -> For working with SAM/BAM files
-3) [Minimap2](https://github.com/lh3/minimap2) -> The aligner used by Dorado
-4) [Modkit](https://github.com/nanoporetech/modkit) -> Extracting summary statistics and read level methylation calls from modified BAM files
-5) [Bedtools](https://github.com/arq5x/bedtools2) -> For generating Bedgraphs from BAM alignment files.
-6) [BedGraphToBigWig](https://genome.ucsc.edu/goldenPath/help/bigWig.html) -> For converting BedGraphs to BigWig files for IGV sessions.
+1) [Dorado](https://github.com/nanoporetech/dorado) -> Basecalling, alignment, demultiplexing.
+2) [Minimap2](https://github.com/lh3/minimap2) -> Alignment if not using dorado.
+3) [Modkit](https://github.com/nanoporetech/modkit) -> Extracting read level methylation metrics from modified BAM files.
 
 ## Modules
 ### Informatics: Processes raw Nanopore/Illumina data from SMF experiments into an AnnData object.
@@ -118,6 +120,9 @@ The following CLI tools need to be installed and configured before using the inf
 
 ## Announcements
 
+### 11/05/25 - Version 0.2.1 is available through PyPI
+Version 0.2.1 makes the core workflow (smftools load) a command line tool that takes in an experiment_config.csv file for input/output and parameter management.
+
 ### 05/29/25 - Version 0.1.6 is available through PyPI.
 Informatics, preprocessing, tools, plotting modules have core functionality that is approaching stability on MacOS(Intel/Silicon) and Linux(Ubuntu). I will work on improving documentation/tutorials shortly. The base PyTorch/Scikit-Learn ML-infrastructure is going through some organizational changes to work with PyTorch Lightning, Hydra, and WanDB to facilitate organizational scaling, multi-device usage, and logging.
 

{smftools-0.1.7 → smftools-0.2.3}/README.md

@@ -9,12 +9,9 @@ While most genomic data structures handle low-coverage data (<100X) along large
 
 ## Dependencies
 The following CLI tools need to be installed and configured before using the informatics (smftools.inform) module of smftools:
-1) [Dorado](https://github.com/nanoporetech/dorado) -> For standard/modified basecalling and alignment. Can be attained by downloading and configuring nanopore MinKnow software.
-2) [Samtools](https://github.com/samtools/samtools) -> For working with SAM/BAM files
-3) [Minimap2](https://github.com/lh3/minimap2) -> The aligner used by Dorado
-4) [Modkit](https://github.com/nanoporetech/modkit) -> Extracting summary statistics and read level methylation calls from modified BAM files
-5) [Bedtools](https://github.com/arq5x/bedtools2) -> For generating Bedgraphs from BAM alignment files.
-6) [BedGraphToBigWig](https://genome.ucsc.edu/goldenPath/help/bigWig.html) -> For converting BedGraphs to BigWig files for IGV sessions.
+1) [Dorado](https://github.com/nanoporetech/dorado) -> Basecalling, alignment, demultiplexing.
+2) [Minimap2](https://github.com/lh3/minimap2) -> Alignment if not using dorado.
+3) [Modkit](https://github.com/nanoporetech/modkit) -> Extracting read level methylation metrics from modified BAM files.
 
 ## Modules
 ### Informatics: Processes raw Nanopore/Illumina data from SMF experiments into an AnnData object.
@@ -29,6 +26,9 @@ The following CLI tools need to be installed and configured before using the inf
 
 ## Announcements
 
+### 11/05/25 - Version 0.2.1 is available through PyPI
+Version 0.2.1 makes the core workflow (smftools load) a command line tool that takes in an experiment_config.csv file for input/output and parameter management.
+
 ### 05/29/25 - Version 0.1.6 is available through PyPI.
 Informatics, preprocessing, tools, plotting modules have core functionality that is approaching stability on MacOS(Intel/Silicon) and Linux(Ubuntu). I will work on improving documentation/tutorials shortly. The base PyTorch/Scikit-Learn ML-infrastructure is going through some organizational changes to work with PyTorch Lightning, Hydra, and WanDB to facilitate organizational scaling, multi-device usage, and logging.
 

smftools-0.2.3/docs/source/basic_usage.md

@@ -0,0 +1,114 @@
+# Basic Usage
+
+## Load Usage
+
+Many use cases for smftools begin here. For most users, the call below will be sufficient to convert any raw SMF dataset from Nanopore/Illumina to an AnnData object:
+
+```shell
+smftools load "/Path_to_experiment_config.csv"
+```
+
+This command takes a user passed config file handling:
+    - I/O pathes (With data input path, FASTA path, optional BED path for subsampling FASTA, and a data output path)
+    - Experiment info (SMF modality, sequencer type, barcoding kit if nanopore, sample sheet with metadata mapping)
+    - Options to override default workflow parameters from smftools/config. Params are handled from default.yaml -> modality_type.yaml -> user passed config.csv.
+
+## Preprocess Usage
+
+This command performs preprocessing on the anndata object. It automatically runs the load command under the hood if starting from raw data.
+
+```shell
+smftools preprocess "/Path_to_experiment_config.csv"
+```
+
+## Spatial Usage
+
+This command performs spatial analysis on the anndata object. It automatically runs the load command and preprocessing under the hood if they have not been already run.
+
+```shell
+smftools spatial "/Path_to_experiment_config.csv"
+```
+
+## HMM Usage
+
+This command performs hmm based feature annotation on the anndata object. It automatically runs the load command and preprocessing under the hood if they have not been already run.
+
+```shell
+smftools hmm "/Path_to_experiment_config.csv"
+```
+
+## Batch Usage
+
+This command performs batch processing of any of the above commands across multiple experiments. It takes in a tsv, txt, or csv of experiment specific config csvs.
+```shell
+smftools batch preprocess "/Path_to_experiment_config_path_list.csv"
+```
+
+## Concatenate Usage
+
+This command concatenates multiple h5ad files and saves them to a new output. The h5ads to concatenate are provided as a txt, tsv, or h5ad file of paths.
+```shell
+smftools concatenate output.h5ad "/Path_to_h5ad_path_list.csv"
+```
+
+## Reading AnnData objects created by smftools
+
+After creating an AnnData object holding your experiment's SMF data, you can load the AnnData object as so:
+
+```
+import smftools as smf
+input_adata = "/Path_to_experiment_AnnData.h5ad.gz"
+adata = safe_read_h5ad(input_adata)
+```
+
+This custom read function will take an optional directory of pickle files for data types that can not normally be saved directly in hdf5 formatting that was saved with the safe_write_h5ad function.
+
+
+If you don't have an AnnData object yet, but want to play with the downstream Preprocessing, Tools, and Plotting modules, you can load a pre-loaded SMF dataset.
+
+Currently, you can do this with our lab's in vitro dCas9 binding kinetics dataset generated from a Hia5 SMF dataset generated with direct m6A high accuracy basecalls:
+
+```
+adata = smf.datasets.dCas9_kinetics()
+adata.obs_names_make_unique()
+```
+
+Alternatively, you can do this with our lab's M.CviPI SMF test data in F1-hybrid natural killer cells generated by NEB EMseq conversion followed by canonical basecalling:
+
+```
+adata = smf.datasets.Kissiov_and_McKenna_2025()
+adata.obs_names_make_unique()
+```
+
+## Writing out AnnData objects to save analysis progress
+
+After preprocessing and downstream analysis of the AnnData object, you can save the AnnData object at any step as so:
+
+```
+import smftools as smf
+from pathlib import Path
+
+output_dir = Path('/Path_to_output_directory')
+output_adata = 'analyzed_adata.h5ad.gz'
+final_output_path = output_dir / output_adata
+safe_write_h5ad(adata, final_output_path, compression='gzip')
+```
+
+This custom save function will make a directory of pickle files for data types that can not normally be saved directly in hdf5 formatting.
+
+## Troubleshooting
+For more advanced usage and help troubleshooting, the API and tutorials for each of the modules is still being developed.
+However, you can currently learn about the functions contained within the module by calling:
+
+```
+smf.inform.__all__
+```
+
+This lists the functions within any given module. If you want to see the associated docstring for a given function, here is an example:
+
+```
+print(smf.inform.load_adata.__doc__)
+```
+
+These docstrings will provide a brief description of the function and also tell you the input parameters and what the function returns.
+In some cases, usage examples will also be provided in the docstring in the form of doctests.

{smftools-0.1.7 → smftools-0.2.3}/docs/source/installation.md

@@ -16,7 +16,7 @@ conda activate smftools
 pip install smftools
 ```
 
-Ensure that you can access dorado, samtools, modkit, bedtools, and BedGraphtoBigWig executables from the terminal in this environment. These are all necessary for the functionality within the Informatics module.
+Ensure that you can access dorado, modkit, and minimap2 executables from the terminal in this environment.
 You may need to add them to $PATH if they are not globally configured.
 For example, if you want to check if dorado is executable, simply run this in the terminal:
 
@@ -24,10 +24,10 @@ For example, if you want to check if dorado is executable, simply run this in th
 dorado
 ```
 
-On Mac OSX, the following can be used to congigure bedtools (with brew) and BedGraphToBigWig (with wget). Change the BedGraphToBigWig link to include the correct architecture for your OS.
+On Mac OSX, the following can be used to congigure minimap2 (with brew) and BedGraphToBigWig (with wget).
 
 ```shell
-brew install bedtools
+brew install minimap2
 wget http://hgdownload.soe.ucsc.edu/admin/exe/macOSX.x86_64/bedGraphToBigWig
 chmod +x bedGraphToBigWig
 sudo mv bedGraphToBigWig /usr/local/bin/
@@ -47,7 +47,10 @@ A python virtual environment can be created as an alternative to conda. I like t
 ```shell
 python -m venv venv-smftools
 source venv-smftools/bin/activate
+pip install --upgrade pip
 pip install .
+pip install ipykernel jupyter
+python -m ipykernel install --user --name=venv-smftools --display-name "Python (smftools)"
 ```
 
 Subsequent use of the installed version of smftools can be run by changing to the smftools directory and activating the venv:

{smftools-0.1.7 → smftools-0.2.3}/experiment_config.csv

@@ -5,15 +5,10 @@ fasta,/path_to_fasta.fasta,Path to initial FASTA file,,str
 fasta_regions_of_interest,/path_to_bed.bed,Path to a bed file to subsample the fasta on.,,str 
 output_directory,/outputs,Directory to act as root for all analysis outputs,,str 
 experiment_name,,An experiment name for the final h5ad file,,str 
+model_dir,/path_to_dorado_model_dir,Path,,str 
 model,None,The dorado basecalling model to use,,str 
 barcode_kit,SQK-NBD114-24,The barcoding kit used for the experiment,,str 
 mapping_threshold,0.05,Minimum proportion of reads mapping to a reference to further use that reference (Ranges from 0-1 as a proportion of mapped reads),,float
-filter_threshold,0.8,Minimum probability to call a canonical base identity,,float
-m6A_threshold,0.8,Minimum probability to flag m6A as True,,float
-m5C_threshold,0.8,Minimum probability to flag m5C as True,,float
-hm5C_threshold,0.8,Minimum probability to flag hm5C as True,,float
 mod_list,[5mC_5hmC],Modified base names for Dorado,"""6mA"", ""5mC_5hmC""",list
 batch_size,4,number of samples to analyze at a time,,int
-conversion_types,[5mC],Types of modification types to use in conversion SMF,"5mC', '6mA'",list
-barcode_both_ends,TRUE,whether to require both ends of a read to be barcoded for demultiplexing,,bool
-trim,FALSE,whether to trim barcodes and adapters from reads during demultiplexing,,bool
+conversion_types,[5mC],Types of modification types to use in conversion SMF,"5mC', '6mA'",list

{smftools-0.1.7 → smftools-0.2.3}/pyproject.toml

@@ -5,7 +5,7 @@ build-backend = "hatchling.build"
 [project]
 name = "smftools"
 description = "Single Molecule Footprinting Analysis in Python."
-requires-python = ">=3.9"
+requires-python = ">=3.9,<3.13"
 license = { file = "LICENSE" }
 authors = [
     {name = "Joseph McKenna"}
@@ -42,6 +42,8 @@ classifiers = [
 dependencies = [
     "anndata>=0.10.0",
     "biopython>=1.79",
+    "captum",
+    "click",
     "fastcluster",
     "hydra-core",
     "igraph",
@@ -53,15 +55,18 @@ dependencies = [
     "omegaconf",
     "pandas>=1.4.2",
     "pod5>=0.1.21",
-    "pomegranate>=1.0.0",
     "pyfaidx>=0.8.0",
+    "pybedtools>=0.12.0",
+    "pyBigWig>=0.3.24",
     "pysam>=0.19.1",
     "scanpy>=1.9",
     "scikit-learn>=1.0.2",
     "scipy>=1.7.3",
+    "shap",
     "seaborn>=0.11",
     "torch>=1.9.0",
     "tqdm",
+    "upsetplot",
     "wandb"
 ]
 dynamic = ["version"]
@@ -70,6 +75,9 @@ dynamic = ["version"]
 Source = "https://github.com/jkmckenna/smftools"
 Documentation = "https://smftools.readthedocs.io/"
 
+[project.scripts]
+smftools = "smftools.cli_entry:cli"
+
 [project.optional-dependencies]
 tests = [
     "pytest",

{smftools-0.1.7 → smftools-0.2.3}/requirements.txt

@@ -1,6 +1,8 @@
 # Essential packages
 anndata>=0.10.0
 biopython>=1.79
+captum
+click
 fastcluster
 hydra-core
 leidenalg
@@ -14,13 +16,16 @@ numpy>=1.22.0,<2
 omegaconf
 pandas>=1.4.2
 pod5>=0.1.21
-pomegranate>=1.0.0
+pybedtools>=0.12.0
+pyBigWig>=0.3.24
 pyfaidx>=0.8.0
 pysam>=0.19.1
-scanpy>=1.9
+scanpy>=1.11
 scikit-learn>=1.0.2
 scipy>=1.7.3
 seaborn>=0.11
+shap
 torch>=1.9.0
 tqdm
+upsetplot
 wandb

{smftools-0.1.7 → smftools-0.2.3}/smftools/__init__.py

@@ -4,12 +4,13 @@ import logging
 import warnings
 
 from . import informatics as inform
+from . import machine_learning as ml
+from . import plotting as pl
 from . import preprocessing as pp
 from . import tools as tl
-from . import plotting as pl
-from . import readwrite, datasets
-from .readwrite import adata_to_df, safe_write_h5ad, merge_barcoded_anndatas
 
+from . import cli, config, datasets, hmm
+from .readwrite import adata_to_df, safe_write_h5ad, safe_read_h5ad, merge_barcoded_anndatas_core
 
 from importlib.metadata import version
 
@@ -19,11 +20,11 @@ __version__ = version(package_name)
 __all__ = [
     "adata_to_df",
     "inform",
+    "ml",
     "pp",
     "tl",
     "pl",
-    "readwrite",
-    "datasets",
+    "datasets"
     "safe_write_h5ad",
-    "merge_barcoded_anndatas"
+    "safe_read_h5ad"
 ]

smftools-0.2.3/smftools/_version.py

@@ -0,0 +1 @@
+__version__ = "0.2.3"

smftools-0.2.3/smftools/cli/cli_flows.py

@@ -0,0 +1,94 @@
+def flow_I(config_path):
+    """
+    High-level function to call for converting raw sequencing data to an adata object. 
+    Command line accesses this through smftools load <config_path>
+    Works for nanopore pod5, fast5, and unaligned modBAM data types for direct SMF workflows.
+    Works for nanopore pod5, fast5, unaligned BAM for conversion SMF workflows.
+    Also works for illumina fastq and unaligned BAM for conversion SMF workflows.
+
+    Parameters:
+        config_path (str): A string representing the file path to the experiment configuration csv file.
+
+    Returns:
+        None
+    """
+    from ..readwrite import safe_read_h5ad, safe_write_h5ad, make_dirs
+    from ..config import LoadExperimentConfig, ExperimentConfig
+    from .load_adata import load_adata
+    from .preprocess_adata import preprocess_adata
+    from .spatial_adata import spatial_adata
+
+    import numpy as np
+    import pandas as pd
+    import anndata as ad
+    import scanpy as sc
+
+    import os
+    from importlib import resources
+    from pathlib import Path
+
+    from datetime import datetime
+    date_str = datetime.today().strftime("%y%m%d")
+    ################################### 1) General params and input organization ###################################
+    # Load experiment config parameters into global variables
+    loader = LoadExperimentConfig(config_path)
+    defaults_dir = resources.files("smftools").joinpath("config")
+    cfg, report = ExperimentConfig.from_var_dict(loader.var_dict, date_str=date_str, defaults_dir=defaults_dir)
+
+    # General config variable init - Necessary user passed inputs
+    smf_modality = cfg.smf_modality # needed for specifying if the data is conversion SMF or direct methylation detection SMF. Or deaminase smf Necessary.
+    input_data_path = Path(cfg.input_data_path)  # Path to a directory of POD5s/FAST5s or to a BAM/FASTQ file. Necessary.
+    output_directory = Path(cfg.output_directory)  # Path to the output directory to make for the analysis. Necessary.
+    fasta = Path(cfg.fasta)  # Path to reference FASTA. Necessary.
+    split_dir = Path(cfg.split_dir) # Relative path to directory for demultiplexing reads
+    split_path = output_directory / split_dir # Absolute path to directory for demultiplexing reads
+
+    # Make initial output directory
+    make_dirs([output_directory])
+
+    bam_suffix = cfg.bam_suffix
+    strands = cfg.strands
+
+    # General config variable init - Optional user passed inputs for enzyme base specificity
+    mod_target_bases = cfg.mod_target_bases  # Nucleobases of interest that may be modified. ['GpC', 'CpG', 'C', 'A']
+
+    # Conversion/deamination specific variable init
+    conversion_types = cfg.conversion_types  # 5mC
+    conversions = cfg.conversions
+
+    # Common Anndata accession params
+    reference_column = cfg.reference_column
+
+    # If conversion_types is passed:
+    if conversion_types:
+        conversions += conversion_types
+
+    ############################################### smftools load start ###############################################
+    initial_adata, initial_adata_path = load_adata(config_path)
+
+    # Initial adata path info
+    initial_backup_dir = initial_adata_path.parent / 'adata_accessory_data'
+    ############################################### smftools load end ###############################################
+
+    ############################################### smftools preprocess start ###############################################
+    pp_adata, pp_adata_path, pp_dedup_adata, pp_dup_rem_adata_path = preprocess_adata(config_path)
+
+    # Preprocessed adata path info
+    pp_adata_basename = initial_adata_path.with_suffix("").name + '_preprocessed.h5ad.gz'
+    pp_adata_path = initial_adata_path / pp_adata_basename
+    pp_backup_dir = pp_adata_path.parent / 'pp_adata_accessory_data'
+
+    # Preprocessed duplicate removed adata path info
+    pp_dup_rem_adata_basename = pp_adata_path.with_suffix("").name + '_duplicates_removed.h5ad.gz'
+    pp_dup_rem_adata_path = pp_adata_path / pp_dup_rem_adata_basename
+    pp_dup_rem_backup_dir= pp_adata_path.parent / 'pp_dup_rem_adata_accessory_data'
+    ############################################### smftools preprocess end ###############################################
+
+    ############################################### smftools spatial start ###############################################
+    # Preprocessed duplicate removed adata with basic analyses appended path info
+    basic_analyzed_adata_basename = pp_dup_rem_adata_path.with_suffix("").name + '_analyzed_I.h5ad.gz'
+    basic_analyzed_adata_path = pp_dup_rem_adata_path / basic_analyzed_adata_basename
+    basic_analyzed_backup_dir= pp_dup_rem_adata_path.parent /'duplicate_removed_analyzed_adata_I_accessory_data'
+
+    spatial_adata, spatial_adata_path = spatial_adata(config_path)
+    ############################################### smftools spatial end ###############################################