yeasti 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- yeasti-0.1.0/LICENSE +7 -0
- yeasti-0.1.0/MANIFEST.in +7 -0
- yeasti-0.1.0/PKG-INFO +169 -0
- yeasti-0.1.0/README.md +152 -0
- yeasti-0.1.0/pyproject.toml +34 -0
- yeasti-0.1.0/setup.cfg +4 -0
- yeasti-0.1.0/src/yeasti/__init__.py +13 -0
- yeasti-0.1.0/src/yeasti/classify.py +102 -0
- yeasti-0.1.0/src/yeasti/construct_read_processing_func.py +35 -0
- yeasti-0.1.0/src/yeasti/data/ENSEMBL_README +120 -0
- yeasti-0.1.0/src/yeasti/data/GFF_README +118 -0
- yeasti-0.1.0/src/yeasti/data/GTF_README +160 -0
- yeasti-0.1.0/src/yeasti/data/Saccharomyces_cerevisiae.R64-1-1.63.chromosome.I.gff3.gz +0 -0
- yeasti-0.1.0/src/yeasti/data/Saccharomyces_cerevisiae.R64-1-1.63.chromosome.I.gtf.gz +0 -0
- yeasti-0.1.0/src/yeasti/data/Saccharomyces_cerevisiae.R64-1-1.dna_sm.chromosome.I.fa.gz +0 -0
- yeasti-0.1.0/src/yeasti/data_path.py +25 -0
- yeasti-0.1.0/src/yeasti/download.py +35 -0
- yeasti-0.1.0/src/yeasti/env.py +49 -0
- yeasti-0.1.0/src/yeasti/init.py +35 -0
- yeasti-0.1.0/src/yeasti/metadata.py +31 -0
- yeasti-0.1.0/src/yeasti/sample_sheet.py +49 -0
- yeasti-0.1.0/src/yeasti/trim.py +61 -0
- yeasti-0.1.0/src/yeasti/version.py +1 -0
- yeasti-0.1.0/src/yeasti/yeasti.py +139 -0
- yeasti-0.1.0/yeasti.egg-info/PKG-INFO +169 -0
- yeasti-0.1.0/yeasti.egg-info/SOURCES.txt +28 -0
- yeasti-0.1.0/yeasti.egg-info/dependency_links.txt +1 -0
- yeasti-0.1.0/yeasti.egg-info/entry_points.txt +2 -0
- yeasti-0.1.0/yeasti.egg-info/requires.txt +2 -0
- yeasti-0.1.0/yeasti.egg-info/top_level.txt +1 -0
yeasti-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
Copyright (c) 2026 Anthony Aylward
|
|
2
|
+
|
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
|
|
4
|
+
|
|
5
|
+
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
|
|
6
|
+
|
|
7
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
yeasti-0.1.0/MANIFEST.in
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
include src/yeasti/data/ENSEMBL_README
|
|
2
|
+
include src/yeasti/data/GFF_README
|
|
3
|
+
include src/yeasti/data/GTF_README
|
|
4
|
+
include src/yeasti/data/Saccharomyces_cerevisiae.R64-1-1.dna_sm.chromosome.I.fa.gz
|
|
5
|
+
include src/yeasti/data/Saccharomyces_cerevisiae.R64-1-1.63.chromosome.I.gff3.gz
|
|
6
|
+
include src/yeasti/data/Saccharomyces_cerevisiae.R64-1-1.63.chromosome.I.gtf.gz
|
|
7
|
+
include src/yeasti/data/metadata.csv
|
yeasti-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,169 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: yeasti
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Small dataset of yeast chromosome I
|
|
5
|
+
Author-email: Anthony Aylward <anthony.aylward@protonmail.com>
|
|
6
|
+
Project-URL: Homepage, https://gitlab.com/aaylward/yeasti
|
|
7
|
+
Project-URL: Documentation, https://aaylward.gitlab.io/yeasti
|
|
8
|
+
Classifier: Programming Language :: Python :: 3
|
|
9
|
+
Classifier: Operating System :: OS Independent
|
|
10
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
11
|
+
Requires-Python: >=3.11
|
|
12
|
+
Description-Content-Type: text/markdown
|
|
13
|
+
License-File: LICENSE
|
|
14
|
+
Requires-Dist: biopython
|
|
15
|
+
Requires-Dist: pysradb
|
|
16
|
+
Dynamic: license-file
|
|
17
|
+
|
|
18
|
+
# YeastI
|
|
19
|
+
|
|
20
|
+
## Introduction
|
|
21
|
+
|
|
22
|
+
This package contains a set of minimal eukaryotic genome and transcriptome datasets from chromosme I of *Saccharomyces cerevisiae*. They may, for example, be useful for rapidly testing a RNA-seq bioinformatics pipeline.
|
|
23
|
+
|
|
24
|
+
## Installation
|
|
25
|
+
|
|
26
|
+
Ensure your computing environment includes Python 3.11 or later and the following bioinformatics tools:
|
|
27
|
+
- sra-tools
|
|
28
|
+
- pigz
|
|
29
|
+
- fastp
|
|
30
|
+
- kraken2
|
|
31
|
+
|
|
32
|
+
Conda is useful for setting up this environment. Then, YeastI can be installed with Pip:
|
|
33
|
+
|
|
34
|
+
```sh
|
|
35
|
+
conda env create -c cond-forge -c bioconda -n yeasti \
|
|
36
|
+
biopython pysradb sra-tools pigz fastp kraken2
|
|
37
|
+
conda activate yeasti
|
|
38
|
+
pip install yeasti
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
## Usage
|
|
42
|
+
|
|
43
|
+
Run `yeasti` without arguments to list the help text of all subcommands
|
|
44
|
+
|
|
45
|
+
```sh
|
|
46
|
+
yeasti
|
|
47
|
+
```
|
|
48
|
+
```
|
|
49
|
+
usage: yeasti [-h] [--version] {init,path,metadata,download,sheet,trim,classify} ...
|
|
50
|
+
|
|
51
|
+
Small dataset of yeast chromosome 1
|
|
52
|
+
|
|
53
|
+
positional arguments:
|
|
54
|
+
{init,path,metadata,download,sheet,trim,classify}
|
|
55
|
+
init fully initialize the dataset by downloading, trimming, and classifying RNA-seq reads
|
|
56
|
+
path show paths to data directory and files
|
|
57
|
+
metadata fetch sra metadata
|
|
58
|
+
download download RNA-seq reads
|
|
59
|
+
sheet sample sheet for yeasti rna-seq dataset
|
|
60
|
+
trim trim downloaded reads
|
|
61
|
+
classify classify downloaded reads
|
|
62
|
+
|
|
63
|
+
options:
|
|
64
|
+
-h, --help show this help message and exit
|
|
65
|
+
--version show program's version number and exit
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
To download and downsample all RNA-seq data, run:
|
|
69
|
+
|
|
70
|
+
```sh
|
|
71
|
+
yeasti init
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
Use `yeasti path` to get the absolute path for a data file
|
|
75
|
+
```sh
|
|
76
|
+
yeasti path data-dir
|
|
77
|
+
yeasti path genome
|
|
78
|
+
yeasti path cds
|
|
79
|
+
yeasti path pep
|
|
80
|
+
yeasti path gff
|
|
81
|
+
yeasti path gtf
|
|
82
|
+
yeasti path sheet
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
To see the sample sheet listing the locations of the RNA-seq reads:
|
|
86
|
+
```sh
|
|
87
|
+
yeasti sheet
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
If you need the data at another location, you can symbolically link to the YeastI dataset, or copy the contents to another directory
|
|
91
|
+
```sh
|
|
92
|
+
ln -s `yeasti path data-dir` test_data/
|
|
93
|
+
cp -r `yeasti path data-dir` test_data/
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
To see the SRA metadata of the RNA-seq reads:
|
|
97
|
+
```sh
|
|
98
|
+
yeasti metadata
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
To execute the initialization steps individually:
|
|
102
|
+
```sh
|
|
103
|
+
yeasti download
|
|
104
|
+
yeasti trim
|
|
105
|
+
yeasti classify
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
## Datasets
|
|
109
|
+

|
|
110
|
+
|
|
111
|
+
There are two groups of datasets:
|
|
112
|
+
1. The directly included package data of **chromosome I** sequence and annotation from assembly R64-1-1 at [Ensembl](https://mart.ensembl.org/Saccharomyces_cerevisiae/Info/Index), including:
|
|
113
|
+
- Genome sequence (FASTA)
|
|
114
|
+
- Coding sequences (FASTA)
|
|
115
|
+
- Peptide sequences (FASTA)
|
|
116
|
+
- Gene annotation (GFF3, GTF)
|
|
117
|
+
2. The RNA-seq reads from [Prigent et al. (2024)](https://www.nature.com/articles/s41467-024-51309-6), which are downloaded from [SRA/NCBI BioProject PRJNA842028](https://www.ncbi.nlm.nih.gov/bioproject/PRJNA842028/) and downsampled to include only chromosome I reads by running an included script.
|
|
118
|
+
- The study sequenced mRNA from *S. cerevisiae* under sulfur starvation at four timepoints with three replicates each, but only two of the timepoints are included in YeastI (T0 and T120).
|
|
119
|
+
- Other similar dataset references can also be found at the [Saccharomyces Genome Database](https://www.yeastgenome.org/).
|
|
120
|
+
|
|
121
|
+
## Downsampling RNA-seq
|
|
122
|
+

|
|
123
|
+
|
|
124
|
+
During the initialization process, after the RNA-seq reads are downloaded they are downsampled by:
|
|
125
|
+
1. Adapter and quality trimming with `fastp`
|
|
126
|
+
2. Classification with `kraken2` to filter out all reads that do not match chromosome I
|
|
127
|
+
|
|
128
|
+
## Configuration
|
|
129
|
+
You can configure some YEASTI settings using environment variables or a TOML file.
|
|
130
|
+
|
|
131
|
+
### Environment variables
|
|
132
|
+
```sh
|
|
133
|
+
export YEASTI_DOWNLOAD_DIR = /absolute/path/to/directory/
|
|
134
|
+
export YEASTI_TAXID = <integer>
|
|
135
|
+
export YEASTI_SAMPLE_SHEET = /absolute/path/to/sample_sheet.csv
|
|
136
|
+
export YEASTI_KRAKEN2_DB /absolute/path/to/kraken2_db/
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
### Config file
|
|
140
|
+
By default, YeastI looks for configuration in TOML format at `$HOME/.yeasti.toml`
|
|
141
|
+
|
|
142
|
+
```toml
|
|
143
|
+
download_dir = /absolute/path/to/directory/
|
|
144
|
+
taxid = <integer>
|
|
145
|
+
sample_sheet = /absolute/path/to/sample_sheet.csv
|
|
146
|
+
kraken2_db = /absolute/path/to/kraken2_db/
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
The location of the config file can also be set using an environment variable:
|
|
150
|
+
```sh
|
|
151
|
+
export YEASTI_CONFIG_FILE=/absolute/path/to/alternate/config.toml
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
## Python API
|
|
155
|
+
|
|
156
|
+
YeastI resources can also be accessed from within python:
|
|
157
|
+
|
|
158
|
+
```python
|
|
159
|
+
import yeasti
|
|
160
|
+
yeasti.init()
|
|
161
|
+
yeasti.path('genome')
|
|
162
|
+
metadata = yeasti.metadata()
|
|
163
|
+
sample_sheet = yeasti.create_sample_sheet(metadata)
|
|
164
|
+
yeasti.write_sample_sheet(sample_sheet, 'sample_sheet.csv')
|
|
165
|
+
yeasti.download()
|
|
166
|
+
yeasti.trim('reads_1.fastq.gz', 'reads_2.fastq.gz')
|
|
167
|
+
yeasti.classify('reads_1.fastq.gz', 'reads_2.fastq.gz')
|
|
168
|
+
```
|
|
169
|
+
|
yeasti-0.1.0/README.md
ADDED
|
@@ -0,0 +1,152 @@
|
|
|
1
|
+
# YeastI
|
|
2
|
+
|
|
3
|
+
## Introduction
|
|
4
|
+
|
|
5
|
+
This package contains a set of minimal eukaryotic genome and transcriptome datasets from chromosme I of *Saccharomyces cerevisiae*. They may, for example, be useful for rapidly testing a RNA-seq bioinformatics pipeline.
|
|
6
|
+
|
|
7
|
+
## Installation
|
|
8
|
+
|
|
9
|
+
Ensure your computing environment includes Python 3.11 or later and the following bioinformatics tools:
|
|
10
|
+
- sra-tools
|
|
11
|
+
- pigz
|
|
12
|
+
- fastp
|
|
13
|
+
- kraken2
|
|
14
|
+
|
|
15
|
+
Conda is useful for setting up this environment. Then, YeastI can be installed with Pip:
|
|
16
|
+
|
|
17
|
+
```sh
|
|
18
|
+
conda env create -c cond-forge -c bioconda -n yeasti \
|
|
19
|
+
biopython pysradb sra-tools pigz fastp kraken2
|
|
20
|
+
conda activate yeasti
|
|
21
|
+
pip install yeasti
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
## Usage
|
|
25
|
+
|
|
26
|
+
Run `yeasti` without arguments to list the help text of all subcommands
|
|
27
|
+
|
|
28
|
+
```sh
|
|
29
|
+
yeasti
|
|
30
|
+
```
|
|
31
|
+
```
|
|
32
|
+
usage: yeasti [-h] [--version] {init,path,metadata,download,sheet,trim,classify} ...
|
|
33
|
+
|
|
34
|
+
Small dataset of yeast chromosome 1
|
|
35
|
+
|
|
36
|
+
positional arguments:
|
|
37
|
+
{init,path,metadata,download,sheet,trim,classify}
|
|
38
|
+
init fully initialize the dataset by downloading, trimming, and classifying RNA-seq reads
|
|
39
|
+
path show paths to data directory and files
|
|
40
|
+
metadata fetch sra metadata
|
|
41
|
+
download download RNA-seq reads
|
|
42
|
+
sheet sample sheet for yeasti rna-seq dataset
|
|
43
|
+
trim trim downloaded reads
|
|
44
|
+
classify classify downloaded reads
|
|
45
|
+
|
|
46
|
+
options:
|
|
47
|
+
-h, --help show this help message and exit
|
|
48
|
+
--version show program's version number and exit
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
To download and downsample all RNA-seq data, run:
|
|
52
|
+
|
|
53
|
+
```sh
|
|
54
|
+
yeasti init
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
Use `yeasti path` to get the absolute path for a data file
|
|
58
|
+
```sh
|
|
59
|
+
yeasti path data-dir
|
|
60
|
+
yeasti path genome
|
|
61
|
+
yeasti path cds
|
|
62
|
+
yeasti path pep
|
|
63
|
+
yeasti path gff
|
|
64
|
+
yeasti path gtf
|
|
65
|
+
yeasti path sheet
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
To see the sample sheet listing the locations of the RNA-seq reads:
|
|
69
|
+
```sh
|
|
70
|
+
yeasti sheet
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
If you need the data at another location, you can symbolically link to the YeastI dataset, or copy the contents to another directory
|
|
74
|
+
```sh
|
|
75
|
+
ln -s `yeasti path data-dir` test_data/
|
|
76
|
+
cp -r `yeasti path data-dir` test_data/
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
To see the SRA metadata of the RNA-seq reads:
|
|
80
|
+
```sh
|
|
81
|
+
yeasti metadata
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
To execute the initialization steps individually:
|
|
85
|
+
```sh
|
|
86
|
+
yeasti download
|
|
87
|
+
yeasti trim
|
|
88
|
+
yeasti classify
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
## Datasets
|
|
92
|
+

|
|
93
|
+
|
|
94
|
+
There are two groups of datasets:
|
|
95
|
+
1. The directly included package data of **chromosome I** sequence and annotation from assembly R64-1-1 at [Ensembl](https://mart.ensembl.org/Saccharomyces_cerevisiae/Info/Index), including:
|
|
96
|
+
- Genome sequence (FASTA)
|
|
97
|
+
- Coding sequences (FASTA)
|
|
98
|
+
- Peptide sequences (FASTA)
|
|
99
|
+
- Gene annotation (GFF3, GTF)
|
|
100
|
+
2. The RNA-seq reads from [Prigent et al. (2024)](https://www.nature.com/articles/s41467-024-51309-6), which are downloaded from [SRA/NCBI BioProject PRJNA842028](https://www.ncbi.nlm.nih.gov/bioproject/PRJNA842028/) and downsampled to include only chromosome I reads by running an included script.
|
|
101
|
+
- The study sequenced mRNA from *S. cerevisiae* under sulfur starvation at four timepoints with three replicates each, but only two of the timepoints are included in YeastI (T0 and T120).
|
|
102
|
+
- Other similar dataset references can also be found at the [Saccharomyces Genome Database](https://www.yeastgenome.org/).
|
|
103
|
+
|
|
104
|
+
## Downsampling RNA-seq
|
|
105
|
+

|
|
106
|
+
|
|
107
|
+
During the initialization process, after the RNA-seq reads are downloaded they are downsampled by:
|
|
108
|
+
1. Adapter and quality trimming with `fastp`
|
|
109
|
+
2. Classification with `kraken2` to filter out all reads that do not match chromosome I
|
|
110
|
+
|
|
111
|
+
## Configuration
|
|
112
|
+
You can configure some YEASTI settings using environment variables or a TOML file.
|
|
113
|
+
|
|
114
|
+
### Environment variables
|
|
115
|
+
```sh
|
|
116
|
+
export YEASTI_DOWNLOAD_DIR = /absolute/path/to/directory/
|
|
117
|
+
export YEASTI_TAXID = <integer>
|
|
118
|
+
export YEASTI_SAMPLE_SHEET = /absolute/path/to/sample_sheet.csv
|
|
119
|
+
export YEASTI_KRAKEN2_DB /absolute/path/to/kraken2_db/
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
### Config file
|
|
123
|
+
By default, YeastI looks for configuration in TOML format at `$HOME/.yeasti.toml`
|
|
124
|
+
|
|
125
|
+
```toml
|
|
126
|
+
download_dir = /absolute/path/to/directory/
|
|
127
|
+
taxid = <integer>
|
|
128
|
+
sample_sheet = /absolute/path/to/sample_sheet.csv
|
|
129
|
+
kraken2_db = /absolute/path/to/kraken2_db/
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
The location of the config file can also be set using an environment variable:
|
|
133
|
+
```sh
|
|
134
|
+
export YEASTI_CONFIG_FILE=/absolute/path/to/alternate/config.toml
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
## Python API
|
|
138
|
+
|
|
139
|
+
YeastI resources can also be accessed from within python:
|
|
140
|
+
|
|
141
|
+
```python
|
|
142
|
+
import yeasti
|
|
143
|
+
yeasti.init()
|
|
144
|
+
yeasti.path('genome')
|
|
145
|
+
metadata = yeasti.metadata()
|
|
146
|
+
sample_sheet = yeasti.create_sample_sheet(metadata)
|
|
147
|
+
yeasti.write_sample_sheet(sample_sheet, 'sample_sheet.csv')
|
|
148
|
+
yeasti.download()
|
|
149
|
+
yeasti.trim('reads_1.fastq.gz', 'reads_2.fastq.gz')
|
|
150
|
+
yeasti.classify('reads_1.fastq.gz', 'reads_2.fastq.gz')
|
|
151
|
+
```
|
|
152
|
+
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["setuptools"]
|
|
3
|
+
build-backend = "setuptools.build_meta"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
name = "yeasti"
|
|
7
|
+
version = "0.1.0"
|
|
8
|
+
authors = [
|
|
9
|
+
{ name="Anthony Aylward", email="anthony.aylward@protonmail.com" },
|
|
10
|
+
]
|
|
11
|
+
description = "Small dataset of yeast chromosome I"
|
|
12
|
+
readme = "README.md"
|
|
13
|
+
requires-python = ">=3.11"
|
|
14
|
+
classifiers = [
|
|
15
|
+
"Programming Language :: Python :: 3",
|
|
16
|
+
"Operating System :: OS Independent",
|
|
17
|
+
"License :: OSI Approved :: MIT License",
|
|
18
|
+
]
|
|
19
|
+
|
|
20
|
+
dependencies = [
|
|
21
|
+
"biopython",
|
|
22
|
+
"pysradb"
|
|
23
|
+
]
|
|
24
|
+
|
|
25
|
+
[project.urls]
|
|
26
|
+
"Homepage" = "https://gitlab.com/aaylward/yeasti"
|
|
27
|
+
"Documentation" = "https://aaylward.gitlab.io/yeasti"
|
|
28
|
+
|
|
29
|
+
[project.scripts]
|
|
30
|
+
yeasti = "yeasti.yeasti:main"
|
|
31
|
+
|
|
32
|
+
[tool.setuptools]
|
|
33
|
+
package-dir = {yeasti="src/yeasti"}
|
|
34
|
+
include-package-data = true
|
yeasti-0.1.0/setup.cfg
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
"""
|
|
2
|
+
yeasti
|
|
3
|
+
"""
|
|
4
|
+
|
|
5
|
+
from yeasti.version import __version__
|
|
6
|
+
from yeasti.metadata import fetch_sra_metadata as metadata
|
|
7
|
+
from yeasti.download import download
|
|
8
|
+
from yeasti.sample_sheet import create_sample_sheet, write_sample_sheet
|
|
9
|
+
from yeasti.data_path import data_path as path
|
|
10
|
+
from yeasti.init import init
|
|
11
|
+
from yeasti.env import YEASTI_TAXID as taxid
|
|
12
|
+
from yeasti.trim import trim
|
|
13
|
+
from yeasti.classify import classify
|
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
#===============================================================================
|
|
2
|
+
# classify.py
|
|
3
|
+
#===============================================================================
|
|
4
|
+
|
|
5
|
+
# Imports ======================================================================
|
|
6
|
+
|
|
7
|
+
import gzip
|
|
8
|
+
import os
|
|
9
|
+
from Bio import SeqIO
|
|
10
|
+
from subprocess import run
|
|
11
|
+
from shutil import which
|
|
12
|
+
from pathlib import Path
|
|
13
|
+
from yeasti.env import CPU_COUNT, YEASTI_KRAKEN2_DB, YEASTI_GENOME, YEASTI_TAXID, YEASTI_DOWNLOAD_DIR
|
|
14
|
+
|
|
15
|
+
# Functions ====================================================================
|
|
16
|
+
|
|
17
|
+
def add_kraken_taxid(seqio_record):
|
|
18
|
+
"""Use kraken2 to classify reads
|
|
19
|
+
|
|
20
|
+
Parameters
|
|
21
|
+
----------
|
|
22
|
+
seqio_record
|
|
23
|
+
Bio.SeqRecord object
|
|
24
|
+
|
|
25
|
+
Returns
|
|
26
|
+
-------
|
|
27
|
+
Bio.SeqRecord
|
|
28
|
+
record with taxid added
|
|
29
|
+
"""
|
|
30
|
+
|
|
31
|
+
seqio_record.id += f'|kraken:taxid|{YEASTI_TAXID}'
|
|
32
|
+
return seqio_record
|
|
33
|
+
|
|
34
|
+
|
|
35
|
+
def classify_kraken2(
|
|
36
|
+
fastq_in_1, fastq_in_2,
|
|
37
|
+
fastq_out_format,
|
|
38
|
+
threads: int = CPU_COUNT
|
|
39
|
+
):
|
|
40
|
+
"""Use kraken2 to classify reads
|
|
41
|
+
|
|
42
|
+
Parameters
|
|
43
|
+
----------
|
|
44
|
+
fastq_in_1, fastq_in_2
|
|
45
|
+
paired-end fastq input files
|
|
46
|
+
fastq_out_format
|
|
47
|
+
format string for fastq output (#.fastq.gz)
|
|
48
|
+
threads : int
|
|
49
|
+
number of threads
|
|
50
|
+
"""
|
|
51
|
+
|
|
52
|
+
if not which('kraken2'):
|
|
53
|
+
raise RuntimeError('kraken2 not found')
|
|
54
|
+
if not Path(YEASTI_KRAKEN2_DB).exists():
|
|
55
|
+
os.mkdir(YEASTI_KRAKEN2_DB)
|
|
56
|
+
run(('k2', 'download-taxonomy', '--db', YEASTI_KRAKEN2_DB), check=True)
|
|
57
|
+
run(('k2', 'add-to-library', '--threads', str(threads), '--no-masking', '--db', YEASTI_KRAKEN2_DB, '--file', str(YEASTI_GENOME)[:-3]), check=True)
|
|
58
|
+
run(('k2', 'build', '--threads', str(threads), '--db', YEASTI_KRAKEN2_DB), check=True)
|
|
59
|
+
run(('k2', 'clean', '--db', YEASTI_KRAKEN2_DB), check=True)
|
|
60
|
+
run((
|
|
61
|
+
'k2', 'classify',
|
|
62
|
+
'--threads', str(threads),
|
|
63
|
+
'--db', YEASTI_KRAKEN2_DB,
|
|
64
|
+
'--quick',
|
|
65
|
+
'--paired',
|
|
66
|
+
'--output', '-',
|
|
67
|
+
'--classified-out', fastq_out_format,
|
|
68
|
+
fastq_in_1,
|
|
69
|
+
fastq_in_2
|
|
70
|
+
), check=True)
|
|
71
|
+
|
|
72
|
+
|
|
73
|
+
def classify(
|
|
74
|
+
fastq_in_1, fastq_in_2,
|
|
75
|
+
temp_dir=YEASTI_DOWNLOAD_DIR,
|
|
76
|
+
threads: int = CPU_COUNT
|
|
77
|
+
):
|
|
78
|
+
"""Use kraken2 to classify reads
|
|
79
|
+
|
|
80
|
+
Parameters
|
|
81
|
+
----------
|
|
82
|
+
fastq_1, fastq_2
|
|
83
|
+
paired-end fastq input files
|
|
84
|
+
temp_dir
|
|
85
|
+
directory for temporary files
|
|
86
|
+
threads : int
|
|
87
|
+
number of cores to use for adapter trimming
|
|
88
|
+
"""
|
|
89
|
+
|
|
90
|
+
with gzip.open(YEASTI_GENOME, 'rt') as f_in, open(str(YEASTI_GENOME)[:-3], 'wt') as f_out:
|
|
91
|
+
SeqIO.write((add_kraken_taxid(r) for r in SeqIO.parse(f_in, 'fasta')), f_out, 'fasta')
|
|
92
|
+
classify_kraken2(
|
|
93
|
+
fastq_in_1,
|
|
94
|
+
fastq_in_2,
|
|
95
|
+
Path(temp_dir) / f'{Path(fastq_in_1).name[:-11]}#.fastq',
|
|
96
|
+
threads=threads
|
|
97
|
+
)
|
|
98
|
+
for fq in fastq_in_1, fastq_in_2:
|
|
99
|
+
with open(fq, 'wb') as f:
|
|
100
|
+
run(('pigz', '--stdout', str(Path(temp_dir) / Path(fq).name)[:-3]), stdout=f, check=True)
|
|
101
|
+
os.remove(str(Path(temp_dir) / Path(fq).name)[:-3])
|
|
102
|
+
os.remove(str(YEASTI_GENOME)[:-3])
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
#===============================================================================
|
|
2
|
+
# construct_read_processing_func.py
|
|
3
|
+
#===============================================================================
|
|
4
|
+
|
|
5
|
+
# Imports ======================================================================
|
|
6
|
+
|
|
7
|
+
from multiprocessing import Pool
|
|
8
|
+
from functools import partial
|
|
9
|
+
from math import floor
|
|
10
|
+
from tempfile import TemporaryDirectory
|
|
11
|
+
from yeasti.env import CPU_COUNT
|
|
12
|
+
|
|
13
|
+
# Functions ====================================================================
|
|
14
|
+
|
|
15
|
+
def construct_read_processing_func(func, parallel=False):
|
|
16
|
+
if parallel:
|
|
17
|
+
def read_processing_func(sample_sheet, processes: int = CPU_COUNT):
|
|
18
|
+
fastq_files = tuple(zip(sample_sheet['fastq_1'], sample_sheet['fastq_2']))
|
|
19
|
+
if processes > 1:
|
|
20
|
+
with Pool(processes=min(processes, len(fastq_files))) as pool, TemporaryDirectory() as temp_dir_name:
|
|
21
|
+
pool.starmap(
|
|
22
|
+
partial(func, temp_dir=temp_dir_name, threads=max(1, floor(processes / len(fastq_files)))),
|
|
23
|
+
fastq_files
|
|
24
|
+
)
|
|
25
|
+
else:
|
|
26
|
+
with TemporaryDirectory() as temp_dir_name:
|
|
27
|
+
for fq1, fq2 in fastq_files:
|
|
28
|
+
func(fq1, fq2, temp_dir=temp_dir_name, threads=processes)
|
|
29
|
+
else:
|
|
30
|
+
def read_processing_func(sample_sheet, processes: int = CPU_COUNT):
|
|
31
|
+
fastq_files = tuple(zip(sample_sheet['fastq_1'], sample_sheet['fastq_2']))
|
|
32
|
+
with TemporaryDirectory() as temp_dir_name:
|
|
33
|
+
for fq1, fq2 in fastq_files:
|
|
34
|
+
func(fq1, fq2, temp_dir=temp_dir_name, threads=processes)
|
|
35
|
+
return read_processing_func
|
|
@@ -0,0 +1,120 @@
|
|
|
1
|
+
#### README ####
|
|
2
|
+
|
|
3
|
+
IMPORTANT: Please note you can download subsets of data via the
|
|
4
|
+
BioMart data mining tool.
|
|
5
|
+
See https://www.ensembl.org/info/data/biomart/ for more information.
|
|
6
|
+
|
|
7
|
+
The genome assembly represented here corresponds to
|
|
8
|
+
GCA_000146045.2
|
|
9
|
+
|
|
10
|
+
#######################
|
|
11
|
+
Fasta DNA dumps
|
|
12
|
+
#######################
|
|
13
|
+
|
|
14
|
+
-----------
|
|
15
|
+
FILE NAMES
|
|
16
|
+
------------
|
|
17
|
+
The files are consistently named following this pattern:
|
|
18
|
+
<species>.<assembly>.<sequence type>.<id type>.<id>.fa.gz
|
|
19
|
+
|
|
20
|
+
<species>: The systematic name of the species.
|
|
21
|
+
<assembly>: The assembly build name.
|
|
22
|
+
<sequence type>:
|
|
23
|
+
* 'dna' - unmasked genomic DNA sequences.
|
|
24
|
+
* 'dna_rm' - masked genomic DNA. Interspersed repeats and low
|
|
25
|
+
complexity regions are detected with the RepeatMasker tool and masked
|
|
26
|
+
by replacing repeats with 'N's.
|
|
27
|
+
* 'dna_sm' - soft-masked genomic DNA. All repeats and low complexity regions
|
|
28
|
+
have been replaced with lowercased versions of their nucleic base
|
|
29
|
+
<id type> One of the following:
|
|
30
|
+
* 'chromosome' - The top-level coordinate system in most species in Ensembl
|
|
31
|
+
* 'nonchromosomal' - Contains DNA that has not been assigned a chromosome
|
|
32
|
+
* 'seqlevel' - This is usually sequence scaffolds, chunks or clones.
|
|
33
|
+
-- 'scaffold' - Larger sequence contigs from the assembly of shorter
|
|
34
|
+
sequencing reads (often from whole genome shotgun, WGS) which could
|
|
35
|
+
not yet be assembled into chromosomes. Often more genome sequencing
|
|
36
|
+
is needed to narrow gaps and establish a tiling path.
|
|
37
|
+
-- 'chunk' - While contig sequences can be assembled into large entities,
|
|
38
|
+
they sometimes have to be artificially broken down into smaller entities
|
|
39
|
+
called 'chunks'. This is due to limitations in the annotation
|
|
40
|
+
pipeline and the finite record size imposed by MySQL which stores the
|
|
41
|
+
sequence and annotation information.
|
|
42
|
+
-- 'clone' - In general this is the smallest sequence entity. It is often
|
|
43
|
+
identical to the sequence of one BAC clone, or sequence region
|
|
44
|
+
of one BAC clone which forms the tiling path.
|
|
45
|
+
<id>: The actual sequence identifier. Depending on the <id type> the <id>
|
|
46
|
+
could represent the name of a chromosome, a scaffold, a contig, a clone ..
|
|
47
|
+
Field is empty for seqlevel files
|
|
48
|
+
fa: All files in these directories represent FASTA database files
|
|
49
|
+
gz: All files are compacted with GNU Zip for storage efficiency.
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
EXAMPLES
|
|
53
|
+
The genomic sequence of human chromosome 1:
|
|
54
|
+
Homo_sapiens.GRCh37.dna.chromosome.1.fa.gz
|
|
55
|
+
|
|
56
|
+
The masked version of the genome sequence on human chromosome 1
|
|
57
|
+
(contains '_rm' or '_sm' in the name):
|
|
58
|
+
Homo_sapiens.GRCh37.dna_rm.chromosome.1.fa.gz
|
|
59
|
+
Homo_sapiens.GRCh37.dna_sm.chromosome.1.fa.gz
|
|
60
|
+
|
|
61
|
+
Non-chromosomal assembly sequences:
|
|
62
|
+
e.g. mitochondrial genome, sequence contigs not yet mapped on chromosomes
|
|
63
|
+
Homo_sapiens.GRCh37.dna.nonchromosomal.fa.gz
|
|
64
|
+
Homo_sapiens.GRCh37.dna_rm.nonchromosomal.fa.gz
|
|
65
|
+
Homo_sapiens.GRCh37.dna_sm.nonchromosomal.fa.gz
|
|
66
|
+
|
|
67
|
+
---------
|
|
68
|
+
TOPLEVEL
|
|
69
|
+
---------
|
|
70
|
+
These files contains all sequence regions flagged as toplevel in an Ensembl
|
|
71
|
+
schema. This includes chromsomes, regions not assembled into chromosomes and
|
|
72
|
+
N padded haplotype/patch regions.
|
|
73
|
+
|
|
74
|
+
EXAMPLES
|
|
75
|
+
|
|
76
|
+
Toplevel sequences unmasked:
|
|
77
|
+
Homo_sapiens.GRCh37.dna.toplevel.fa.gz
|
|
78
|
+
|
|
79
|
+
Toplevel soft/hard masked sequences:
|
|
80
|
+
Homo_sapiens.GRCh37.dna_sm.toplevel.fa.gz
|
|
81
|
+
Homo_sapiens.GRCh37.dna_rm.toplevel.fa.gz
|
|
82
|
+
|
|
83
|
+
-----------------
|
|
84
|
+
PRIMARY ASSEMBLY
|
|
85
|
+
-----------------
|
|
86
|
+
Primary assembly contains all toplevel sequence regions excluding haplotypes
|
|
87
|
+
and patches. This file is best used for performing sequence similarity searches
|
|
88
|
+
where patch and haplotype sequences would confuse analysis. If the primary
|
|
89
|
+
assembly file is not present, that indicates that there are no haplotype/patch
|
|
90
|
+
regions, and the 'toplevel' file is equivalent.
|
|
91
|
+
|
|
92
|
+
EXAMPLES
|
|
93
|
+
|
|
94
|
+
Primary assembly sequences unmasked:
|
|
95
|
+
Homo_sapiens.GRCh37.dna.primary_assembly.fa.gz
|
|
96
|
+
|
|
97
|
+
Primary assembly soft/hard masked sequences:
|
|
98
|
+
Homo_sapiens.GRCh37.dna_sm.primary_assembly.fa.gz
|
|
99
|
+
Homo_sapiens.GRCh37.dna_rm.primary_assembly.fa.gz
|
|
100
|
+
|
|
101
|
+
--------------
|
|
102
|
+
SPECIAL CASES
|
|
103
|
+
--------------
|
|
104
|
+
Some chromosomes have alternate haplotypes which are presented in files with
|
|
105
|
+
the haplotype sequence only:
|
|
106
|
+
Homo_sapiens.GRCh37.dna_rm.chromosome.HSCHR6_MHC_QBL.fa.gz
|
|
107
|
+
Homo_sapiens.GRCh37.dna_rm.chromosome.HSCHR17_1.fa.gz
|
|
108
|
+
|
|
109
|
+
All alternative assembly and patch regions have their sequence padded
|
|
110
|
+
with N's to ensure alignment programs can report the correct index
|
|
111
|
+
regions
|
|
112
|
+
|
|
113
|
+
e.g. A patch region with a start position of 1,000,001 will have 1e6 N's added
|
|
114
|
+
its start so an alignment program will report coordinates with respect to the
|
|
115
|
+
whole chromosome.
|
|
116
|
+
|
|
117
|
+
Human has sequenced Y chromosomes and the pseudoautosomal region (PAR)
|
|
118
|
+
on the Y is annotated. By definition the PAR region is identical on the
|
|
119
|
+
X and Y chromosome. The Y chromosome file contains the Y chromsome
|
|
120
|
+
minus these repeated PAR regions i.e. the unique portion of Y.
|