smftools 0.1.0__tar.gz → 0.1.3__tar.gz

{smftools-0.1.0 → smftools-0.1.3}/.gitignore

@@ -1,10 +1,9 @@
 # Python
 __pycache__/
-/src/smftools/_version.py
 
 # Build files
 build/
-dist/
+/dist/
 /hatch.toml
 /Pipfile
 /Pipfile.lock
@@ -16,30 +15,36 @@ dist/
 /*-venv/
 /env-*/
 /venv-*/
+venv/
 /environment.yml
 
+# Tests
+/tests/_test_inputs/
+/tests/_test_outputs/
+
 # OS
-*.DS_Store
-*.LSOverride
-*Thumbs.db
+.DS_Store
+.LSOverride
+Thumbs.db
 *.ipynb_checkpoints/
 *.directory
 
 # IDEs and editors
-*.vscode/
-*.idea/
+.vscode/
+.idea/
 *.iml
 
 # Logs
 *.log
 
 # temp files
-temp/
-tmp/
+/temp*
+/tmp*
+*.temp
+*.tmp
 
 # Coverage reports
 .coverage
 htmlcov/
 
 # Docs
-/docs

smftools-0.1.3/.readthedocs.yaml

@@ -0,0 +1,17 @@
+version: 2
+build:
+  os: ubuntu-20.04
+  tools:
+    python: "3.12"
+sphinx:
+  configuration: docs/source/conf.py
+  fail_on_warning: true
+python:
+  install:
+    - method: pip
+      path: .
+      extra_requirements:
+      - docs
+submodules:
+  include: all
+  recursive: true

smftools-0.1.3/CONTRIBUTING.md

@@ -0,0 +1,3 @@
+Contributing
+============
+Contributions to smftools are not currently being reviewed or accepted due to the pre-alpha phase status of the project. More mature versions of the project will have contribution guidelines added.

smftools-0.1.3/PKG-INFO

@@ -0,0 +1,94 @@
+Metadata-Version: 2.3
+Name: smftools
+Version: 0.1.3
+Summary: Single Molecule Footprinting Analysis in Python.
+Project-URL: Source, https://github.com/jkmckenna/smftools
+Project-URL: Documentation, https://smftools.readthedocs.io/
+Author: Joseph McKenna
+Maintainer-email: Joseph McKenna <jkmckenna@berkeley.edu>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: anndata,chromatin-accessibility,machine-learning,nanopore,protein-dna-binding,single-locus,single-molecule-footprinting
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Natural Language :: English
+Classifier: Operating System :: MacOS :: MacOS X
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Classifier: Topic :: Scientific/Engineering :: Visualization
+Requires-Python: >=3.9
+Requires-Dist: anndata>=0.10.0
+Requires-Dist: biopython>=1.79
+Requires-Dist: cython>=0.29.28
+Requires-Dist: networkx>=3.2
+Requires-Dist: numpy<2,>=1.22.0
+Requires-Dist: pandas>=1.4.2
+Requires-Dist: pod5>=0.1.21
+Requires-Dist: pomegranate>1.0.0
+Requires-Dist: pyfaidx>=0.8.0
+Requires-Dist: pysam>=0.19.1
+Requires-Dist: scanpy>=1.9
+Requires-Dist: scikit-learn>=1.0.2
+Requires-Dist: scipy>=1.7.3
+Requires-Dist: seaborn>=0.11
+Requires-Dist: torch>=1.9.0
+Requires-Dist: tqdm
+Provides-Extra: docs
+Requires-Dist: ipython>=7.20; extra == 'docs'
+Requires-Dist: matplotlib!=3.6.1; extra == 'docs'
+Requires-Dist: myst-nb>=1; extra == 'docs'
+Requires-Dist: myst-parser>=2; extra == 'docs'
+Requires-Dist: nbsphinx>=0.9; extra == 'docs'
+Requires-Dist: readthedocs-sphinx-search; extra == 'docs'
+Requires-Dist: setuptools; extra == 'docs'
+Requires-Dist: sphinx-autodoc-typehints>=1.25.2; extra == 'docs'
+Requires-Dist: sphinx-book-theme>=1.1.0; extra == 'docs'
+Requires-Dist: sphinx-copybutton; extra == 'docs'
+Requires-Dist: sphinx-design; extra == 'docs'
+Requires-Dist: sphinx>=7; extra == 'docs'
+Requires-Dist: sphinxcontrib-bibtex; extra == 'docs'
+Requires-Dist: sphinxext-opengraph; extra == 'docs'
+Provides-Extra: tests
+Requires-Dist: pytest; extra == 'tests'
+Requires-Dist: pytest-cov; extra == 'tests'
+Description-Content-Type: text/markdown
+
+[![PyPI](https://img.shields.io/pypi/v/smftools.svg)](https://pypi.org/project/smftools)
+[![Docs](https://readthedocs.org/projects/smftools/badge/?version=latest)](https://smftools.readthedocs.io/en/latest/?badge=latest)
+
+# smftools
+A Python tool for processing raw sequencing data derived from single molecule footprinting experiments into [anndata](https://anndata.readthedocs.io/en/latest/) objects. Additional functionality for preprocessing, analysis, and visualization.
+
+## Philosophy
+While most genomic data structures handle low-coverage data (<100X) along large references, smftools prioritizes high-coverage data (scalable to at least 1 million X coverage) of a few genomic loci at a time. This enables efficient data storage, rapid data operations, hierarchical metadata handling, seamless integration with various machine-learning packages, and ease of visualization. Furthermore, functionality is modularized, enabling analysis sessions to be saved, reloaded, and easily shared with collaborators. Analyses are centered around the [anndata](https://anndata.readthedocs.io/en/latest/) object, and are heavily inspired by the work conducted within the single-cell genomics community.
+
+## Dependencies
+The following CLI tools need to be installed and configured before using the informatics (smftools.inform) module of smftools:
+1) [Dorado](https://github.com/nanoporetech/dorado) -> For standard/modified basecalling and alignment. Can be attained by downloading and configuring nanopore MinKnow software.
+2) [Samtools](https://github.com/samtools/samtools) -> For working with SAM/BAM files
+3) [Minimap2](https://github.com/lh3/minimap2) -> The aligner used by Dorado
+4) [Modkit](https://github.com/nanoporetech/modkit) -> Extracting summary statistics and read level methylation calls from modified BAM files
+5) [Bedtools](https://github.com/arq5x/bedtools2) -> For generating Bedgraphs from BAM alignment files.
+6) [BedGraphToBigWig](https://genome.ucsc.edu/goldenPath/help/bigWig.html) -> For converting BedGraphs to BigWig files for IGV sessions.
+
+## Modules
+### Informatics: Processes raw Nanopore/Illumina data from SMF experiments into an AnnData object.
+![](docs/source/_static/smftools_informatics_diagram.png)
+### Preprocessing: Appends QC metrics to the AnnData object and perfroms filtering.
+![](docs/source/_static/smftools_preprocessing_diagram.png)
+- Tools: Appends various analyses to the AnnData object.
+- Plotting: Visualization of analyses stored within the AnnData object.
+
+## Announcements
+### 09/09/24 - The pre-alpha phase package ([smftools-0.1.1](https://pypi.org/project/smftools/))
+The informatics module has been bumped to alpha-phase status. This module can deal with POD5s and unaligned BAMS from nanopore conversion and direct SMF experiments, as well as FASTQs from Illumina conversion SMF experiments. Primary output from this module is an AnnData object containing all relevant SMF data, which is compatible with all downstream smftools modules. The other modules are still in pre-alpha phase. Preprocessing, Tools, and Plotting modules should be promoted to alpha-phase within the next month or so.
+
+### 08/30/24 - The pre-alpha phase package ([smftools-0.1.0](https://pypi.org/project/smftools/)) is installable through pypi!
+Currently, this package (smftools-0.1.0) is going through rapid improvement (dependency handling accross Linux and Mac OS, testing, documentation, debugging) and is still too early in development for standard use. The underlying functionality was originally developed as a collection of scripts for single molecule footprinting (SMF) experiments in our lab, but is being packaged/developed to facilitate the expansion of SMF to any lab that is interested in performing these styles of experiments/analyses. The alpha-phase package is expected to be available within a couple months, so stay tuned!

smftools-0.1.3/README.md

@@ -0,0 +1,32 @@
+[![PyPI](https://img.shields.io/pypi/v/smftools.svg)](https://pypi.org/project/smftools)
+[![Docs](https://readthedocs.org/projects/smftools/badge/?version=latest)](https://smftools.readthedocs.io/en/latest/?badge=latest)
+
+# smftools
+A Python tool for processing raw sequencing data derived from single molecule footprinting experiments into [anndata](https://anndata.readthedocs.io/en/latest/) objects. Additional functionality for preprocessing, analysis, and visualization.
+
+## Philosophy
+While most genomic data structures handle low-coverage data (<100X) along large references, smftools prioritizes high-coverage data (scalable to at least 1 million X coverage) of a few genomic loci at a time. This enables efficient data storage, rapid data operations, hierarchical metadata handling, seamless integration with various machine-learning packages, and ease of visualization. Furthermore, functionality is modularized, enabling analysis sessions to be saved, reloaded, and easily shared with collaborators. Analyses are centered around the [anndata](https://anndata.readthedocs.io/en/latest/) object, and are heavily inspired by the work conducted within the single-cell genomics community.
+
+## Dependencies
+The following CLI tools need to be installed and configured before using the informatics (smftools.inform) module of smftools:
+1) [Dorado](https://github.com/nanoporetech/dorado) -> For standard/modified basecalling and alignment. Can be attained by downloading and configuring nanopore MinKnow software.
+2) [Samtools](https://github.com/samtools/samtools) -> For working with SAM/BAM files
+3) [Minimap2](https://github.com/lh3/minimap2) -> The aligner used by Dorado
+4) [Modkit](https://github.com/nanoporetech/modkit) -> Extracting summary statistics and read level methylation calls from modified BAM files
+5) [Bedtools](https://github.com/arq5x/bedtools2) -> For generating Bedgraphs from BAM alignment files.
+6) [BedGraphToBigWig](https://genome.ucsc.edu/goldenPath/help/bigWig.html) -> For converting BedGraphs to BigWig files for IGV sessions.
+
+## Modules
+### Informatics: Processes raw Nanopore/Illumina data from SMF experiments into an AnnData object.
+![](docs/source/_static/smftools_informatics_diagram.png)
+### Preprocessing: Appends QC metrics to the AnnData object and perfroms filtering.
+![](docs/source/_static/smftools_preprocessing_diagram.png)
+- Tools: Appends various analyses to the AnnData object.
+- Plotting: Visualization of analyses stored within the AnnData object.
+
+## Announcements
+### 09/09/24 - The pre-alpha phase package ([smftools-0.1.1](https://pypi.org/project/smftools/))
+The informatics module has been bumped to alpha-phase status. This module can deal with POD5s and unaligned BAMS from nanopore conversion and direct SMF experiments, as well as FASTQs from Illumina conversion SMF experiments. Primary output from this module is an AnnData object containing all relevant SMF data, which is compatible with all downstream smftools modules. The other modules are still in pre-alpha phase. Preprocessing, Tools, and Plotting modules should be promoted to alpha-phase within the next month or so.
+
+### 08/30/24 - The pre-alpha phase package ([smftools-0.1.0](https://pypi.org/project/smftools/)) is installable through pypi!
+Currently, this package (smftools-0.1.0) is going through rapid improvement (dependency handling accross Linux and Mac OS, testing, documentation, debugging) and is still too early in development for standard use. The underlying functionality was originally developed as a collection of scripts for single molecule footprinting (SMF) experiments in our lab, but is being packaged/developed to facilitate the expansion of SMF to any lab that is interested in performing these styles of experiments/analyses. The alpha-phase package is expected to be available within a couple months, so stay tuned!

smftools-0.1.3/docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

smftools-0.1.3/docs/make.bat

@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=source
+set BUILDDIR=build
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.https://www.sphinx-doc.org/
+	exit /b 1
+)
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd

smftools-0.1.3/docs/source/_templates/tmp

@@ -0,0 +1 @@
+

smftools-0.1.3/docs/source/api/datasets.md

@@ -0,0 +1,9 @@
+## Datasets:
+
+```{eval-rst}
+.. module:: smftools.datasets
+```
+
+```{eval-rst}
+.. currentmodule:: smftools
+```

smftools-0.1.3/docs/source/api/index.md

@@ -0,0 +1,26 @@
+# API
+
+Import smftools as:
+
+```
+import smftools as smf
+```
+
+```{toctree}
+:maxdepth: 2
+
+informatics
+preprocessing
+tools
+datasets
+```
+
+## Informatics module diagram
+```{image} ../_static/smftools_informatics_diagram.png
+:width: 800px
+```
+
+## Preprocessing module diagram
+```{image} ../_static/smftools_preprocessing_diagram.png
+:width: 800px
+```

smftools-0.1.3/docs/source/api/informatics.md

@@ -0,0 +1,27 @@
+## Informatics: `inform`
+
+## Informatics module diagram
+```{image} ../_static/smftools_informatics_diagram.png
+:width: 1000px
+```
+
+```{eval-rst}
+.. module:: smftools.inform
+```
+
+```{eval-rst}
+.. currentmodule:: smftools
+```
+
+Processes raw sequencing data to load an adata object.
+
+
+### Diagram of final steps of Direct SMF workflow
+```{image} ../_static/modkit_extract_to_adata.png
+:width: 1000px
+```
+
+### Diagram of final steps of Conversion SMF workflow
+```{image} ../_static/converted_BAM_to_adata.png
+:width: 1000px
+```

smftools-0.1.3/docs/source/api/preprocessing.md

@@ -0,0 +1,14 @@
+## Preprocessing: `pp`
+
+## Preprocessing module diagram
+```{image} ../_static/smftools_preprocessing_diagram.png
+:width: 1000px
+```
+
+```{eval-rst}
+.. module:: smftools.pp
+```
+
+```{eval-rst}
+.. currentmodule:: smftools
+```

smftools-0.1.3/docs/source/api/tools.md

@@ -0,0 +1,9 @@
+## Tools: `tl`
+
+```{eval-rst}
+.. module:: smftools.tl
+```
+
+```{eval-rst}
+.. currentmodule:: smftools
+```

smftools-0.1.3/docs/source/basic_usage.md

@@ -0,0 +1,75 @@
+# Basic Usage
+
+Import SmfTools:
+
+```
+import smftools as smf
+```
+
+## Informatics Module Usage
+
+Many use cases for smftools begin here. For most users, the call below will be sufficient to convert any raw SMF dataset to an AnnData object:
+
+```
+config_path = "/Path_to_experiment_config.csv"
+smf.inform.load_adata(config_path)
+```
+
+## Loading AnnData objects created by the informatics module
+
+After creating an AnnData object holding your experiment's SMF data, you can load the AnnData object as so:
+
+```
+import anndata as ad
+input_adata = "/Path_to_experiment_AnnData.h5ad.gz"
+adata = ad.read_h5ad(input_file)
+adata.obs_names_make_unique()
+```
+
+If you don't have an AnnData object yet, but want to play with the downstream Preprocessing, Tools, and Plotting modules, you can load a pre-loaded SMF dataset.
+
+Currently, you can do this with our lab's in vitro dCas9 binding kinetics dataset generated from a Hia5 SMF dataset generated with direct m6A high accuracy basecalls:
+
+```
+adata = smf.datasets.dCas9_kinetics()
+adata.obs_names_make_unique()
+```
+
+Alternatively, you can do this with our lab's M.CviPI SMF test data in F1-hybrid natural killer cells generated by NEB EMseq conversion followed by canonical basecalling:
+
+```
+adata = smf.datasets.Kissiov_and_McKenna_2025()
+adata.obs_names_make_unique()
+```
+
+## Writing out AnnData objects to save analysis progress
+
+After preprocessing and downstream analysis of the AnnData object, you can save the AnnData object at any step as so:
+
+```
+import anndata as ad
+import os
+
+output_dir = '/Path_to_output_directory'
+output_adata = 'analyzed_adata.h5ad.gz'
+final_output = os.path.join(output_dir, output_adata)
+adata.write_h5ad(final_output, compression='gzip')
+```
+
+
+## Troubleshooting
+For more advanced usage and help troubleshooting, the API and tutorials for each of the modules is still being developed.
+However, you can currently learn about the functions contained within the module by calling:
+
+```
+smf.inform.__all__
+```
+
+This lists the functions within any given module. If you want to see the associated docstring for a given function, here is an example:
+
+```
+print(smf.inform.load_adata.__doc__)
+```
+
+These docstrings will provide a brief description of the function and also tell you the input parameters and what the function returns.
+In some cases, usage examples will also be provided in the docstring in the form of doctests.

smftools-0.1.3/docs/source/conf.py

@@ -0,0 +1,117 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# For the full list of built-in configuration values, see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+import sys
+import os
+from pathlib import Path
+HERE = Path(__file__).parent
+PARENT_PARENT_HERE = HERE.parents[1]
+SRC_PATH = PARENT_PARENT_HERE / 'src'
+sys.path.insert(0, str(SRC_PATH))
+for x in os.walk(str(SRC_PATH)):
+  sys.path.insert(0, x[0])
+print(sys.path)
+try:
+    import smftools
+    print("smftools imported successfully.")
+except ImportError:
+    print("smftools is not imported.")
+# -- Project information -----------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
+
+project = 'smftools'
+copyright = '2024, Joseph McKenna'
+author = 'Joseph McKenna'
+release = '0.1.0'
+repository_url = 'https://github.com/jkmckenna/smftools'
+
+# -- General configuration ---------------------------------------------------
+# Bibliography settings
+bibtex_bibfiles = ["references.bib"]
+bibtex_reference_style = "author_year"
+nitpicky = True
+needs_sphinx = "4.0"
+
+master_doc = "index"
+templates_path = ['_templates']
+exclude_patterns = [
+    "_build",
+    "Thumbs.db",
+    ".DS_Store",
+    # exclude version md files
+    "release-notes/[!i]*.md"
+]
+extensions = [
+    "myst_nb",
+    "sphinx_copybutton",
+    "sphinx.ext.autodoc",
+    "sphinx.ext.intersphinx",
+    "sphinx.ext.doctest",
+    "sphinx.ext.coverage",
+    "sphinx.ext.mathjax",
+    "sphinx.ext.napoleon",
+    "sphinx.ext.autosummary",
+    "sphinx.ext.extlinks",
+    "sphinxcontrib.bibtex",
+    "matplotlib.sphinxext.plot_directive",
+    "sphinx_autodoc_typehints",
+    "sphinx_design",
+    "sphinx_search.extension",
+    "sphinxext.opengraph",
+]
+
+# Generate the API documentation when building
+autosummary_generate = True
+autodoc_member_order = "bysource"
+napoleon_google_docstring = True
+napoleon_numpy_docstring = False
+napoleon_include_init_with_doc = False
+napoleon_use_rtype = True  # having a separate entry generally helps readability
+napoleon_use_param = True
+napoleon_custom_sections = [("Params", "Parameters")]
+todo_include_todos = False
+api_dir = HERE / "api" 
+myst_enable_extensions = [
+    "amsmath",
+    "colon_fence",
+    "deflist",
+    "dollarmath",
+    "html_image",
+    "html_admonition",
+]
+myst_url_schemes = ("http", "https", "mailto", "ftp")
+myst_heading_anchors = 3
+nb_output_stderr = "remove"
+nb_execution_mode = "off"
+nb_merge_streams = True
+
+suppress_warnings = [
+    "myst.header"
+]
+
+typehints_defaults = "braces"
+
+# html_context = {
+#     "display_github": True,
+#     "github_user": "jkmckenna",
+#     "github_repo": project,
+#     "github_version": "main",
+#     "conf_py_path": "/docs/source/",
+# }
+
+# -- Options for HTML output -------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
+
+html_theme = "sphinx_book_theme"
+html_title = project
+
+html_theme_options = {
+    "repository_url": repository_url,
+    "use_repository_button": True,
+    "show_toc_level": 1,
+    "path_to_docs": "docs/",
+    "repository_branch": release,
+}
+
+html_static_path = ['_static']

smftools-0.1.3/docs/source/contributors.md

@@ -0,0 +1,9 @@
+# Contributors
+
+## Current
+
+- Joseph Mckenna, lead developer (2024)
+
+## Acknowledgments
+
+- [Tjian/Darzacq Lab](https://www.tjian-darzacq.mcb.berkeley.edu/): Funding and a supportive lab environment!

smftools-0.1.3/docs/source/dev/index.md

@@ -0,0 +1,3 @@
+(contribution-guide)=
+
+# Contributing

smftools-0.1.3/docs/source/index.md

@@ -0,0 +1,54 @@
+smftools documentation
+======================
+
+```{include} ../../README.md
+:end-before: '## Dependencies'
+```
+
+::::{grid} 1 2 2 2
+:gutter: 2
+
+:::{grid-item-card} Installation {octicon}`plug;1em;`
+:link: installation
+:link-type: doc
+
+smftools installation instructions
+:::
+
+:::{grid-item-card} Tutorials {octicon}`play;1em;`
+:link: tutorials/index
+:link-type: doc
+
+Jupyter notebook tutorial of smftools usage.
+:::
+
+:::{grid-item-card} API reference {octicon}`book;1em;`
+:link: api/index
+:link-type: doc
+
+The API reference contains a detailed description of
+the smftools API.
+:::
+
+:::{grid-item-card} GitHub {octicon}`mark-github;1em;`
+:link: https://github.com/jkmckenna/smftools
+
+smftools GitHub link
+:::
+::::
+
+```{toctree}
+:hidden: true
+:maxdepth: 1
+
+installation
+basic_usage
+tutorials/index
+api/index
+release-notes/index
+dev/index
+contributors
+references
+```
+
+[github]: https://github.com/jkmckenna/smftools