hyphlow 1.0.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- hyphlow-1.0.0/PKG-INFO +345 -0
- hyphlow-1.0.0/README.md +330 -0
- hyphlow-1.0.0/hyphlow/__init__.py +0 -0
- hyphlow-1.0.0/hyphlow/common_ui.py +910 -0
- hyphlow-1.0.0/hyphlow/common_utils.py +152 -0
- hyphlow-1.0.0/hyphlow/main.py +967 -0
- hyphlow-1.0.0/hyphlow/t1_st1_logic.py +344 -0
- hyphlow-1.0.0/hyphlow/t1_st1_ui.py +888 -0
- hyphlow-1.0.0/hyphlow/t1_st2_logic.py +212 -0
- hyphlow-1.0.0/hyphlow/t1_st3_logic.py +217 -0
- hyphlow-1.0.0/hyphlow/t1_st4_logic.py +247 -0
- hyphlow-1.0.0/hyphlow/t1_st4_ui.py +209 -0
- hyphlow-1.0.0/hyphlow/t1_st5_logic.py +473 -0
- hyphlow-1.0.0/hyphlow/t1_st5_ui.py +643 -0
- hyphlow-1.0.0/hyphlow/t2_tagging_logic.py +479 -0
- hyphlow-1.0.0/hyphlow/t2_tagging_ui.py +1115 -0
- hyphlow-1.0.0/hyphlow/t3_hyphy_logic.py +286 -0
- hyphlow-1.0.0/hyphlow/t3_hyphy_ui.py +1234 -0
- hyphlow-1.0.0/hyphlow/t4_summary_logic.py +300 -0
- hyphlow-1.0.0/hyphlow/t4_summary_ui.py +342 -0
- hyphlow-1.0.0/hyphlow.egg-info/PKG-INFO +345 -0
- hyphlow-1.0.0/hyphlow.egg-info/SOURCES.txt +26 -0
- hyphlow-1.0.0/hyphlow.egg-info/dependency_links.txt +1 -0
- hyphlow-1.0.0/hyphlow.egg-info/entry_points.txt +2 -0
- hyphlow-1.0.0/hyphlow.egg-info/requires.txt +7 -0
- hyphlow-1.0.0/hyphlow.egg-info/top_level.txt +1 -0
- hyphlow-1.0.0/pyproject.toml +29 -0
- hyphlow-1.0.0/setup.cfg +4 -0
hyphlow-1.0.0/PKG-INFO
ADDED
|
@@ -0,0 +1,345 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: hyphlow
|
|
3
|
+
Version: 1.0.0
|
|
4
|
+
Summary: Bioinformatics selection analysis pipeline and GUI
|
|
5
|
+
Author: Hyejung Kwon, Ryan K Schott
|
|
6
|
+
Requires-Python: >=3.8
|
|
7
|
+
Description-Content-Type: text/markdown
|
|
8
|
+
Requires-Dist: PyQt5
|
|
9
|
+
Requires-Dist: qtawesome
|
|
10
|
+
Requires-Dist: pandas
|
|
11
|
+
Requires-Dist: ete3
|
|
12
|
+
Requires-Dist: rapidfuzz
|
|
13
|
+
Requires-Dist: taxopy
|
|
14
|
+
Requires-Dist: xlsxwriter
|
|
15
|
+
|
|
16
|
+
# <img src="assets/schott_lab_logo.png" width="40" align="top"> <img src="assets/logo.png" width="50" align="top"> Schott_lab_HYphlow
|
|
17
|
+
|
|
18
|
+
<img src="assets/banner.png" alt="HYphlow Banner" width="60%" align="left">
|
|
19
|
+
<br clear="left"/>
|
|
20
|
+
|
|
21
|
+
## Table of Contents
|
|
22
|
+
* [Overview](#overview)
|
|
23
|
+
* [Data Preparation](#data-preparation)
|
|
24
|
+
* [Species Label Standardization](#species-label-standardization)
|
|
25
|
+
* [Tree Pruning](#tree-pruning)
|
|
26
|
+
* [Data Reconciliation](#data-reconciliation)
|
|
27
|
+
* [Tree Annotation](#tree-annotation)
|
|
28
|
+
* [HyPhy Execution](#hyphy-execution)
|
|
29
|
+
* [Results Summary](#results-summary)
|
|
30
|
+
* [Supported HyPhy Models](#supported-hyphy-models)
|
|
31
|
+
* [Dependencies](#dependencies)
|
|
32
|
+
|
|
33
|
+
---
|
|
34
|
+
|
|
35
|
+
## Overview
|
|
36
|
+
|
|
37
|
+
HYphlow is a streamlined workflow for preparing input files, annotating foreground branches, running batch HyPhy analyses, and summarizing results for multiple genes.
|
|
38
|
+
|
|
39
|
+
---
|
|
40
|
+
|
|
41
|
+
## Data Preparation
|
|
42
|
+
|
|
43
|
+
### Species Label Standardization
|
|
44
|
+
The Species Label Standardization module checks species names in CSV metadata against the NCBI taxonomy database and standardizes FASTA and Newick labels into a consistent format.
|
|
45
|
+
|
|
46
|
+
CSV metadata files are used to validate species names using NCBI taxonomy. FASTA headers and Newick tree labels are then formatted to remove extra information, such as accession numbers and trailing tags, extracting clean and consistent species or subspecies labels.
|
|
47
|
+
|
|
48
|
+
**Key Features**
|
|
49
|
+
* **Compares species names** in the CSV metadata file with the NCBI taxonomy database.
|
|
50
|
+
* **Identifies valid, similar, missing, or unmatched** species names.
|
|
51
|
+
* **Supports automatic correction** for similar species names when possible.
|
|
52
|
+
* **Formats FASTA headers** by using the gene name to truncate unnecessary strings, retaining clean species/subspecies labels.
|
|
53
|
+
* **Formats Newick tree leaf names** into a consistent species/subspecies structure.
|
|
54
|
+
* **Automatically recognizes trinomial nomenclature** to support and preserve valid subspecies names.
|
|
55
|
+
* **Removes extra accession numbers** (e.g., XM_, NM_) or unnecessary label information.
|
|
56
|
+
* **Generates standardized files** with `_fmt` tags and detailed Excel validation reports for downstream analysis.
|
|
57
|
+
|
|
58
|
+
**Input**
|
|
59
|
+
Use a CSV metadata file, FASTA alignment file, and Newick tree file.
|
|
60
|
+
```text
|
|
61
|
+
trait_metadata.csv
|
|
62
|
+
gene_alignment.fasta
|
|
63
|
+
gene_tree.nwk
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Example CSV metadata:
|
|
67
|
+
```text
|
|
68
|
+
species,trait
|
|
69
|
+
Terrapene carolina,terrestrial
|
|
70
|
+
Chrysemys picta bellii,freshwater
|
|
71
|
+
Chelonia mydas,marine
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
Example FASTA headers before standardization:
|
|
75
|
+
```text
|
|
76
|
+
>Terrapene_carolina_OPN3_XM_024203557
|
|
77
|
+
ATGCGT...
|
|
78
|
+
>Chrysemys_picta_bellii_OPN3_XM_005301234
|
|
79
|
+
ATGCGT...
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Example FASTA headers after standardization:
|
|
83
|
+
```text
|
|
84
|
+
>Terrapene_carolina
|
|
85
|
+
ATGCGT...
|
|
86
|
+
>Chrysemys_picta_bellii
|
|
87
|
+
ATGCGT...
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
**Output**
|
|
91
|
+
The module generates validated metadata, standardized FASTA files, standardized Newick files, and detailed Excel reports.
|
|
92
|
+
```text
|
|
93
|
+
trait_metadata_fmt_v1_MMDD.csv
|
|
94
|
+
gene_alignment_fmt_v1_MMDD.fasta
|
|
95
|
+
gene_tree_fmt_v1_MMDD.nwk
|
|
96
|
+
Rpt_trait_metadata_fmt_v1_MMDD.xlsx
|
|
97
|
+
Rpt_gene_alignment_fmt_v1_MMDD.xlsx
|
|
98
|
+
Rpt_gene_tree_fmt_v1_MMDD.xlsx
|
|
99
|
+
```
|
|
100
|
+
These standardized files can be used directly in the Tree Pruning, Data Reconciliation, and Tree Annotation modules.
|
|
101
|
+
|
|
102
|
+
### Tree Pruning
|
|
103
|
+
The Tree Pruning module generates gene-specific Newick trees by pruning a master tree based on the taxa present in each FASTA alignment file.
|
|
104
|
+
|
|
105
|
+
When running phylogenetic analyses for multiple genes, each gene alignment may contain a different set of species. This module reduces manual work by automatically removing taxa that are not present in the FASTA file, producing a matching tree for each alignment.
|
|
106
|
+
|
|
107
|
+
**Key Features**
|
|
108
|
+
* **Uses a master Newick tree** as the reference tree.
|
|
109
|
+
* **Reads species labels** from FASTA alignment files.
|
|
110
|
+
* **Removes tree leaves** that are not present in the FASTA file.
|
|
111
|
+
* **Generates gene-specific** pruned Newick trees.
|
|
112
|
+
* **Checks whether FASTA taxa** are present in the master tree.
|
|
113
|
+
* **Reports missing or unmatched taxa** to prevent execution errors.
|
|
114
|
+
* **Saves pruned trees** for downstream HyPhy analyses.
|
|
115
|
+
|
|
116
|
+
**Input**
|
|
117
|
+
Use a master Newick tree and one or more FASTA alignment files.
|
|
118
|
+
For FASTA files, it is recommended to use the formatted outputs generated from the Standardization step.
|
|
119
|
+
```text
|
|
120
|
+
Master_Species_Tree.nwk
|
|
121
|
+
GeneName_aln_fmt_v1_MMDD.fasta
|
|
122
|
+
AnotherGene_aln_fmt_v1_MMDD.fasta
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
Example FASTA file:
|
|
126
|
+
```text
|
|
127
|
+
>Species_A
|
|
128
|
+
ATGCGT...
|
|
129
|
+
>Species_B
|
|
130
|
+
ATGCGT...
|
|
131
|
+
>Species_C
|
|
132
|
+
ATGCGT...
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
Example master tree:
|
|
136
|
+
```text
|
|
137
|
+
(Species_A,Species_B,Species_C,Species_D,Species_E);
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
**Output**
|
|
141
|
+
The module generates pruned Newick trees that match the taxa in each FASTA alignment, along with detailed Excel reports.
|
|
142
|
+
```text
|
|
143
|
+
GeneName_prn_v1_MMDD.nwk
|
|
144
|
+
AnotherGene_prn_v1_MMDD.nwk
|
|
145
|
+
Rpt_GeneName_prn_v1_MMDD.xlsx
|
|
146
|
+
Rpt_AnotherGene_prn_v1_MMDD.xlsx
|
|
147
|
+
```
|
|
148
|
+
The pruned trees can be used directly in the Data Reconciliation and Tree Annotation modules.
|
|
149
|
+
|
|
150
|
+
### Data Reconciliation
|
|
151
|
+
The Data Reconciliation module checks and standardizes species labels across CSV, FASTA, and Newick tree files.
|
|
152
|
+
|
|
153
|
+
When preparing comparative or phylogenetic analyses, species names often appear in different formats across input files. For example, the same species may appear with extra sequence IDs in FASTA headers, different spellings in CSV files, or inconsistent labels in Newick trees. This module helps identify and correct these mismatches before running downstream analyses.
|
|
154
|
+
|
|
155
|
+
**Key Features**
|
|
156
|
+
* **Checks species names** in CSV files against the NCBI taxonomy database.
|
|
157
|
+
* **Standardizes FASTA headers** into species-level labels.
|
|
158
|
+
* **Standardizes Newick tree leaf names** into species-level labels.
|
|
159
|
+
* **Compares species labels** across CSV, FASTA, and Newick files.
|
|
160
|
+
* **Identifies missing or mismatched taxa** between input files.
|
|
161
|
+
* **Supports automatic correction** for similar species names when possible.
|
|
162
|
+
* **Generates cleaned input files** for downstream analysis.
|
|
163
|
+
* **Saves validation and reconciliation reports** for review.
|
|
164
|
+
|
|
165
|
+
**Input**
|
|
166
|
+
Use CSV, FASTA, and Newick tree files that contain overlapping species labels.
|
|
167
|
+
For FASTA and NWK files, it is recommended to use the formatted or pruned outputs generated from the previous Data Preparation steps.
|
|
168
|
+
```text
|
|
169
|
+
trait_data.csv
|
|
170
|
+
GeneName_aln_fmt_v1_MMDD.fasta
|
|
171
|
+
GeneName_prn_v1_MMDD.nwk
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
The CSV file should include a species column.
|
|
175
|
+
```text
|
|
176
|
+
species,trait
|
|
177
|
+
Species_A,nocturnal
|
|
178
|
+
Species_B,diurnal
|
|
179
|
+
Species_C,nocturnal
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
The FASTA file may contain longer sequence headers.
|
|
183
|
+
```text
|
|
184
|
+
>Species_A_gene1
|
|
185
|
+
ATGCGT...
|
|
186
|
+
>Species_B_gene1
|
|
187
|
+
ATGCGT...
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
The Newick tree should contain matching species labels.
|
|
191
|
+
```text
|
|
192
|
+
(Species_A,Species_B,Species_C);
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
**Output**
|
|
196
|
+
The module generates standardized files and detailed Excel reconciliation reports.
|
|
197
|
+
```text
|
|
198
|
+
GeneName_aln_rec_v1_MMDD.fasta
|
|
199
|
+
GeneName_tree_rec_v1_MMDD.nwk
|
|
200
|
+
Rpt_GeneName_Reconciliation_Details_MMDD.xlsx
|
|
201
|
+
```
|
|
202
|
+
These cleaned files can be used directly in the Tree Annotation and HyPhy Execution modules.
|
|
203
|
+
|
|
204
|
+
---
|
|
205
|
+
|
|
206
|
+
## Tree Annotation
|
|
207
|
+
The Tree Annotation module uses CSV trait data to automatically add foreground labels (`{FG}`) to a Newick tree file.
|
|
208
|
+
|
|
209
|
+
When preparing HyPhy analyses, users often need to manually decide which branches should be treated as foreground branches. This module helps automate that step by comparing trait information across the tree and generating a foreground-annotated Newick file based on ancestral state reconstruction.
|
|
210
|
+
|
|
211
|
+
**Key Features**
|
|
212
|
+
* **Uses CSV trait data** and a matching Newick tree file automatically.
|
|
213
|
+
* **Allows users to choose** the target trait columns and foreground phenotypic values.
|
|
214
|
+
* **Identifies candidate foreground branches** using Fitch, Sankoff, and Felsenstein ML algorithms.
|
|
215
|
+
* **Generates scalable preview images (SVG)** for each method and the final consensus.
|
|
216
|
+
* **Saves a foreground-annotated** Newick file ready for HyPhy execution.
|
|
217
|
+
* **Exports a detailed CSV report** scoring internal nodes.
|
|
218
|
+
|
|
219
|
+
**Supported Annotation Methods**
|
|
220
|
+
Currently supported methods include:
|
|
221
|
+
* Fitch parsimony
|
|
222
|
+
* Sankoff parsimony
|
|
223
|
+
* Felsenstein likelihood
|
|
224
|
+
* Strict Consensus
|
|
225
|
+
|
|
226
|
+
**Input**
|
|
227
|
+
Use a trait CSV file and a matching reconciled Newick tree file:
|
|
228
|
+
```text
|
|
229
|
+
trait_data.csv
|
|
230
|
+
GeneName_tree_rec_v1_MMDD.nwk
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
Example CSV format:
|
|
234
|
+
```text
|
|
235
|
+
species,trait
|
|
236
|
+
Species_A,nocturnal
|
|
237
|
+
Species_B,nocturnal
|
|
238
|
+
Species_C,diurnal
|
|
239
|
+
Species_D,diurnal
|
|
240
|
+
```
|
|
241
|
+
|
|
242
|
+
**Output**
|
|
243
|
+
The module generates an annotated Newick tree, preview images, and a report:
|
|
244
|
+
```text
|
|
245
|
+
GeneName_tree_annotated_Strict_Consensus.nwk
|
|
246
|
+
GeneName_tree_annotated_figure.svg
|
|
247
|
+
Rpt_GeneName_tree_annotated_MMDD.csv
|
|
248
|
+
```
|
|
249
|
+
The annotated Newick tree can be used directly in the HyPhy Execution module.
|
|
250
|
+
|
|
251
|
+
---
|
|
252
|
+
|
|
253
|
+
## HyPhy Execution
|
|
254
|
+
The HyPhy Execution module generates and runs batch HyPhy analysis scripts using matched FASTA alignment files and Newick tree files.
|
|
255
|
+
|
|
256
|
+
When analyzing multiple genes, users often need to prepare separate HyPhy commands for each alignment and tree pair. This module reduces that manual work by matching input files, generating execution scripts, and supporting parallel HyPhy analyses.
|
|
257
|
+
|
|
258
|
+
**Key Features**
|
|
259
|
+
* **Matches FASTA alignment files** with corresponding Newick tree files automatically.
|
|
260
|
+
* **Supports batch execution** of multiple HyPhy analyses in parallel.
|
|
261
|
+
* **Supports both all-branch** analysis and foreground-branch analysis.
|
|
262
|
+
* **Generates an editable** HyPhy execution bash script.
|
|
263
|
+
* **Allows users to adjust CPU/thread** settings before running analyses.
|
|
264
|
+
* **Checks input files** before execution and reports potential file-matching issues.
|
|
265
|
+
* **Saves HyPhy output files** and detailed log reports for downstream review.
|
|
266
|
+
|
|
267
|
+
**Input**
|
|
268
|
+
Use matched FASTA alignment files and Newick tree files:
|
|
269
|
+
```text
|
|
270
|
+
GeneName_aln_rec_v1_MMDD.fasta
|
|
271
|
+
GeneName_tree_rec_v1_MMDD.nwk
|
|
272
|
+
```
|
|
273
|
+
|
|
274
|
+
For foreground-branch analyses, use Newick tree files that contain foreground branch labels (`{FG}`):
|
|
275
|
+
```text
|
|
276
|
+
GeneName_tree_annotated_Strict_Consensus.nwk
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
**Output**
|
|
280
|
+
The module generates HyPhy JSON result files and individual log reports:
|
|
281
|
+
```text
|
|
282
|
+
GeneName_aln_rec_v1_MMDD_BUSTED.json
|
|
283
|
+
GeneName_aln_rec_v1_MMDD_BUSTED_log.txt
|
|
284
|
+
```
|
|
285
|
+
|
|
286
|
+
---
|
|
287
|
+
|
|
288
|
+
## Results Summary
|
|
289
|
+
The Results Summary module extracts key results from HyPhy `.json` output files and saves them into a single Excel summary file.
|
|
290
|
+
|
|
291
|
+
When running HyPhy analyses for multiple genes, users often need to open many JSON files manually to check p-values, LRT scores, and significant branches. This module reduces that manual work by collecting the main results automatically.
|
|
292
|
+
|
|
293
|
+
**Key Features**
|
|
294
|
+
* **Drag and drop** HyPhy `.json` result files directly into the interface.
|
|
295
|
+
* **Detects the HyPhy model** used for each file automatically.
|
|
296
|
+
* **Extracts model-specific** summary results efficiently.
|
|
297
|
+
* **Saves all extracted results** into a single Excel workbook.
|
|
298
|
+
* **Organizes results** into separate sheets by HyPhy model.
|
|
299
|
+
* **Highlights significant results** automatically (e.g., p-value < 0.05).
|
|
300
|
+
* **Shows parsing errors** or skipped files clearly in the log console.
|
|
301
|
+
|
|
302
|
+
**Input**
|
|
303
|
+
Use the final `.json` output files generated by the HyPhy Execution module:
|
|
304
|
+
```text
|
|
305
|
+
GeneName_aln_rec_v1_MMDD_BUSTED.json
|
|
306
|
+
GeneName_aln_rec_v1_MMDD_RELAX.json
|
|
307
|
+
```
|
|
308
|
+
|
|
309
|
+
**Output**
|
|
310
|
+
The module generates a single organized Excel workbook:
|
|
311
|
+
```text
|
|
312
|
+
HyPhy_results_summary_MMDD.xlsx
|
|
313
|
+
```
|
|
314
|
+
|
|
315
|
+
---
|
|
316
|
+
|
|
317
|
+
## Supported HyPhy Models
|
|
318
|
+
Currently supported models across all modules include:
|
|
319
|
+
* BUSTED
|
|
320
|
+
* aBSREL
|
|
321
|
+
* RELAX
|
|
322
|
+
* FEL
|
|
323
|
+
* MEME
|
|
324
|
+
* FUBAR
|
|
325
|
+
* SLAC
|
|
326
|
+
|
|
327
|
+
---
|
|
328
|
+
|
|
329
|
+
## Dependencies
|
|
330
|
+
To use the full HYphlow pipeline, you can set up your environment using either Conda (recommended) or Pip.
|
|
331
|
+
|
|
332
|
+
**Option 1: Using Conda **
|
|
333
|
+
The easiest way to install all dependencies, including the HyPhy engine and Python packages, is to use the provided `environment.yml` file. This creates a dedicated virtual environment.
|
|
334
|
+
```bash
|
|
335
|
+
conda env create -f environment.yml
|
|
336
|
+
conda activate hyphlow_env
|
|
337
|
+
hyphy --version
|
|
338
|
+
```
|
|
339
|
+
|
|
340
|
+
**Option 2: Using Pip**
|
|
341
|
+
If you already have the HyPhy engine installed on your system and prefer not to use Conda, you can manually install the required Python GUI and bioinformatics packages using `requirements.txt`.
|
|
342
|
+
```bash
|
|
343
|
+
# Ensure HyPhy is installed first: conda install -c bioconda hyphy
|
|
344
|
+
pip install -r requirements.txt
|
|
345
|
+
```
|
hyphlow-1.0.0/README.md
ADDED
|
@@ -0,0 +1,330 @@
|
|
|
1
|
+
# <img src="assets/schott_lab_logo.png" width="40" align="top"> <img src="assets/logo.png" width="50" align="top"> Schott_lab_HYphlow
|
|
2
|
+
|
|
3
|
+
<img src="assets/banner.png" alt="HYphlow Banner" width="60%" align="left">
|
|
4
|
+
<br clear="left"/>
|
|
5
|
+
|
|
6
|
+
## Table of Contents
|
|
7
|
+
* [Overview](#overview)
|
|
8
|
+
* [Data Preparation](#data-preparation)
|
|
9
|
+
* [Species Label Standardization](#species-label-standardization)
|
|
10
|
+
* [Tree Pruning](#tree-pruning)
|
|
11
|
+
* [Data Reconciliation](#data-reconciliation)
|
|
12
|
+
* [Tree Annotation](#tree-annotation)
|
|
13
|
+
* [HyPhy Execution](#hyphy-execution)
|
|
14
|
+
* [Results Summary](#results-summary)
|
|
15
|
+
* [Supported HyPhy Models](#supported-hyphy-models)
|
|
16
|
+
* [Dependencies](#dependencies)
|
|
17
|
+
|
|
18
|
+
---
|
|
19
|
+
|
|
20
|
+
## Overview
|
|
21
|
+
|
|
22
|
+
HYphlow is a streamlined workflow for preparing input files, annotating foreground branches, running batch HyPhy analyses, and summarizing results for multiple genes.
|
|
23
|
+
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
## Data Preparation
|
|
27
|
+
|
|
28
|
+
### Species Label Standardization
|
|
29
|
+
The Species Label Standardization module checks species names in CSV metadata against the NCBI taxonomy database and standardizes FASTA and Newick labels into a consistent format.
|
|
30
|
+
|
|
31
|
+
CSV metadata files are used to validate species names using NCBI taxonomy. FASTA headers and Newick tree labels are then formatted to remove extra information, such as accession numbers and trailing tags, extracting clean and consistent species or subspecies labels.
|
|
32
|
+
|
|
33
|
+
**Key Features**
|
|
34
|
+
* **Compares species names** in the CSV metadata file with the NCBI taxonomy database.
|
|
35
|
+
* **Identifies valid, similar, missing, or unmatched** species names.
|
|
36
|
+
* **Supports automatic correction** for similar species names when possible.
|
|
37
|
+
* **Formats FASTA headers** by using the gene name to truncate unnecessary strings, retaining clean species/subspecies labels.
|
|
38
|
+
* **Formats Newick tree leaf names** into a consistent species/subspecies structure.
|
|
39
|
+
* **Automatically recognizes trinomial nomenclature** to support and preserve valid subspecies names.
|
|
40
|
+
* **Removes extra accession numbers** (e.g., XM_, NM_) or unnecessary label information.
|
|
41
|
+
* **Generates standardized files** with `_fmt` tags and detailed Excel validation reports for downstream analysis.
|
|
42
|
+
|
|
43
|
+
**Input**
|
|
44
|
+
Use a CSV metadata file, FASTA alignment file, and Newick tree file.
|
|
45
|
+
```text
|
|
46
|
+
trait_metadata.csv
|
|
47
|
+
gene_alignment.fasta
|
|
48
|
+
gene_tree.nwk
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
Example CSV metadata:
|
|
52
|
+
```text
|
|
53
|
+
species,trait
|
|
54
|
+
Terrapene carolina,terrestrial
|
|
55
|
+
Chrysemys picta bellii,freshwater
|
|
56
|
+
Chelonia mydas,marine
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
Example FASTA headers before standardization:
|
|
60
|
+
```text
|
|
61
|
+
>Terrapene_carolina_OPN3_XM_024203557
|
|
62
|
+
ATGCGT...
|
|
63
|
+
>Chrysemys_picta_bellii_OPN3_XM_005301234
|
|
64
|
+
ATGCGT...
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
Example FASTA headers after standardization:
|
|
68
|
+
```text
|
|
69
|
+
>Terrapene_carolina
|
|
70
|
+
ATGCGT...
|
|
71
|
+
>Chrysemys_picta_bellii
|
|
72
|
+
ATGCGT...
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
**Output**
|
|
76
|
+
The module generates validated metadata, standardized FASTA files, standardized Newick files, and detailed Excel reports.
|
|
77
|
+
```text
|
|
78
|
+
trait_metadata_fmt_v1_MMDD.csv
|
|
79
|
+
gene_alignment_fmt_v1_MMDD.fasta
|
|
80
|
+
gene_tree_fmt_v1_MMDD.nwk
|
|
81
|
+
Rpt_trait_metadata_fmt_v1_MMDD.xlsx
|
|
82
|
+
Rpt_gene_alignment_fmt_v1_MMDD.xlsx
|
|
83
|
+
Rpt_gene_tree_fmt_v1_MMDD.xlsx
|
|
84
|
+
```
|
|
85
|
+
These standardized files can be used directly in the Tree Pruning, Data Reconciliation, and Tree Annotation modules.
|
|
86
|
+
|
|
87
|
+
### Tree Pruning
|
|
88
|
+
The Tree Pruning module generates gene-specific Newick trees by pruning a master tree based on the taxa present in each FASTA alignment file.
|
|
89
|
+
|
|
90
|
+
When running phylogenetic analyses for multiple genes, each gene alignment may contain a different set of species. This module reduces manual work by automatically removing taxa that are not present in the FASTA file, producing a matching tree for each alignment.
|
|
91
|
+
|
|
92
|
+
**Key Features**
|
|
93
|
+
* **Uses a master Newick tree** as the reference tree.
|
|
94
|
+
* **Reads species labels** from FASTA alignment files.
|
|
95
|
+
* **Removes tree leaves** that are not present in the FASTA file.
|
|
96
|
+
* **Generates gene-specific** pruned Newick trees.
|
|
97
|
+
* **Checks whether FASTA taxa** are present in the master tree.
|
|
98
|
+
* **Reports missing or unmatched taxa** to prevent execution errors.
|
|
99
|
+
* **Saves pruned trees** for downstream HyPhy analyses.
|
|
100
|
+
|
|
101
|
+
**Input**
|
|
102
|
+
Use a master Newick tree and one or more FASTA alignment files.
|
|
103
|
+
For FASTA files, it is recommended to use the formatted outputs generated from the Standardization step.
|
|
104
|
+
```text
|
|
105
|
+
Master_Species_Tree.nwk
|
|
106
|
+
GeneName_aln_fmt_v1_MMDD.fasta
|
|
107
|
+
AnotherGene_aln_fmt_v1_MMDD.fasta
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
Example FASTA file:
|
|
111
|
+
```text
|
|
112
|
+
>Species_A
|
|
113
|
+
ATGCGT...
|
|
114
|
+
>Species_B
|
|
115
|
+
ATGCGT...
|
|
116
|
+
>Species_C
|
|
117
|
+
ATGCGT...
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
Example master tree:
|
|
121
|
+
```text
|
|
122
|
+
(Species_A,Species_B,Species_C,Species_D,Species_E);
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
**Output**
|
|
126
|
+
The module generates pruned Newick trees that match the taxa in each FASTA alignment, along with detailed Excel reports.
|
|
127
|
+
```text
|
|
128
|
+
GeneName_prn_v1_MMDD.nwk
|
|
129
|
+
AnotherGene_prn_v1_MMDD.nwk
|
|
130
|
+
Rpt_GeneName_prn_v1_MMDD.xlsx
|
|
131
|
+
Rpt_AnotherGene_prn_v1_MMDD.xlsx
|
|
132
|
+
```
|
|
133
|
+
The pruned trees can be used directly in the Data Reconciliation and Tree Annotation modules.
|
|
134
|
+
|
|
135
|
+
### Data Reconciliation
|
|
136
|
+
The Data Reconciliation module checks and standardizes species labels across CSV, FASTA, and Newick tree files.
|
|
137
|
+
|
|
138
|
+
When preparing comparative or phylogenetic analyses, species names often appear in different formats across input files. For example, the same species may appear with extra sequence IDs in FASTA headers, different spellings in CSV files, or inconsistent labels in Newick trees. This module helps identify and correct these mismatches before running downstream analyses.
|
|
139
|
+
|
|
140
|
+
**Key Features**
|
|
141
|
+
* **Checks species names** in CSV files against the NCBI taxonomy database.
|
|
142
|
+
* **Standardizes FASTA headers** into species-level labels.
|
|
143
|
+
* **Standardizes Newick tree leaf names** into species-level labels.
|
|
144
|
+
* **Compares species labels** across CSV, FASTA, and Newick files.
|
|
145
|
+
* **Identifies missing or mismatched taxa** between input files.
|
|
146
|
+
* **Supports automatic correction** for similar species names when possible.
|
|
147
|
+
* **Generates cleaned input files** for downstream analysis.
|
|
148
|
+
* **Saves validation and reconciliation reports** for review.
|
|
149
|
+
|
|
150
|
+
**Input**
|
|
151
|
+
Use CSV, FASTA, and Newick tree files that contain overlapping species labels.
|
|
152
|
+
For FASTA and NWK files, it is recommended to use the formatted or pruned outputs generated from the previous Data Preparation steps.
|
|
153
|
+
```text
|
|
154
|
+
trait_data.csv
|
|
155
|
+
GeneName_aln_fmt_v1_MMDD.fasta
|
|
156
|
+
GeneName_prn_v1_MMDD.nwk
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
The CSV file should include a species column.
|
|
160
|
+
```text
|
|
161
|
+
species,trait
|
|
162
|
+
Species_A,nocturnal
|
|
163
|
+
Species_B,diurnal
|
|
164
|
+
Species_C,nocturnal
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
The FASTA file may contain longer sequence headers.
|
|
168
|
+
```text
|
|
169
|
+
>Species_A_gene1
|
|
170
|
+
ATGCGT...
|
|
171
|
+
>Species_B_gene1
|
|
172
|
+
ATGCGT...
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
The Newick tree should contain matching species labels.
|
|
176
|
+
```text
|
|
177
|
+
(Species_A,Species_B,Species_C);
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
**Output**
|
|
181
|
+
The module generates standardized files and detailed Excel reconciliation reports.
|
|
182
|
+
```text
|
|
183
|
+
GeneName_aln_rec_v1_MMDD.fasta
|
|
184
|
+
GeneName_tree_rec_v1_MMDD.nwk
|
|
185
|
+
Rpt_GeneName_Reconciliation_Details_MMDD.xlsx
|
|
186
|
+
```
|
|
187
|
+
These cleaned files can be used directly in the Tree Annotation and HyPhy Execution modules.
|
|
188
|
+
|
|
189
|
+
---
|
|
190
|
+
|
|
191
|
+
## Tree Annotation
|
|
192
|
+
The Tree Annotation module uses CSV trait data to automatically add foreground labels (`{FG}`) to a Newick tree file.
|
|
193
|
+
|
|
194
|
+
When preparing HyPhy analyses, users often need to manually decide which branches should be treated as foreground branches. This module helps automate that step by comparing trait information across the tree and generating a foreground-annotated Newick file based on ancestral state reconstruction.
|
|
195
|
+
|
|
196
|
+
**Key Features**
|
|
197
|
+
* **Uses CSV trait data** and a matching Newick tree file automatically.
|
|
198
|
+
* **Allows users to choose** the target trait columns and foreground phenotypic values.
|
|
199
|
+
* **Identifies candidate foreground branches** using Fitch, Sankoff, and Felsenstein ML algorithms.
|
|
200
|
+
* **Generates scalable preview images (SVG)** for each method and the final consensus.
|
|
201
|
+
* **Saves a foreground-annotated** Newick file ready for HyPhy execution.
|
|
202
|
+
* **Exports a detailed CSV report** scoring internal nodes.
|
|
203
|
+
|
|
204
|
+
**Supported Annotation Methods**
|
|
205
|
+
Currently supported methods include:
|
|
206
|
+
* Fitch parsimony
|
|
207
|
+
* Sankoff parsimony
|
|
208
|
+
* Felsenstein likelihood
|
|
209
|
+
* Strict Consensus
|
|
210
|
+
|
|
211
|
+
**Input**
|
|
212
|
+
Use a trait CSV file and a matching reconciled Newick tree file:
|
|
213
|
+
```text
|
|
214
|
+
trait_data.csv
|
|
215
|
+
GeneName_tree_rec_v1_MMDD.nwk
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
Example CSV format:
|
|
219
|
+
```text
|
|
220
|
+
species,trait
|
|
221
|
+
Species_A,nocturnal
|
|
222
|
+
Species_B,nocturnal
|
|
223
|
+
Species_C,diurnal
|
|
224
|
+
Species_D,diurnal
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
**Output**
|
|
228
|
+
The module generates an annotated Newick tree, preview images, and a report:
|
|
229
|
+
```text
|
|
230
|
+
GeneName_tree_annotated_Strict_Consensus.nwk
|
|
231
|
+
GeneName_tree_annotated_figure.svg
|
|
232
|
+
Rpt_GeneName_tree_annotated_MMDD.csv
|
|
233
|
+
```
|
|
234
|
+
The annotated Newick tree can be used directly in the HyPhy Execution module.
|
|
235
|
+
|
|
236
|
+
---
|
|
237
|
+
|
|
238
|
+
## HyPhy Execution
|
|
239
|
+
The HyPhy Execution module generates and runs batch HyPhy analysis scripts using matched FASTA alignment files and Newick tree files.
|
|
240
|
+
|
|
241
|
+
When analyzing multiple genes, users often need to prepare separate HyPhy commands for each alignment and tree pair. This module reduces that manual work by matching input files, generating execution scripts, and supporting parallel HyPhy analyses.
|
|
242
|
+
|
|
243
|
+
**Key Features**
|
|
244
|
+
* **Matches FASTA alignment files** with corresponding Newick tree files automatically.
|
|
245
|
+
* **Supports batch execution** of multiple HyPhy analyses in parallel.
|
|
246
|
+
* **Supports both all-branch** analysis and foreground-branch analysis.
|
|
247
|
+
* **Generates an editable** HyPhy execution bash script.
|
|
248
|
+
* **Allows users to adjust CPU/thread** settings before running analyses.
|
|
249
|
+
* **Checks input files** before execution and reports potential file-matching issues.
|
|
250
|
+
* **Saves HyPhy output files** and detailed log reports for downstream review.
|
|
251
|
+
|
|
252
|
+
**Input**
|
|
253
|
+
Use matched FASTA alignment files and Newick tree files:
|
|
254
|
+
```text
|
|
255
|
+
GeneName_aln_rec_v1_MMDD.fasta
|
|
256
|
+
GeneName_tree_rec_v1_MMDD.nwk
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
For foreground-branch analyses, use Newick tree files that contain foreground branch labels (`{FG}`):
|
|
260
|
+
```text
|
|
261
|
+
GeneName_tree_annotated_Strict_Consensus.nwk
|
|
262
|
+
```
|
|
263
|
+
|
|
264
|
+
**Output**
|
|
265
|
+
The module generates HyPhy JSON result files and individual log reports:
|
|
266
|
+
```text
|
|
267
|
+
GeneName_aln_rec_v1_MMDD_BUSTED.json
|
|
268
|
+
GeneName_aln_rec_v1_MMDD_BUSTED_log.txt
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
---
|
|
272
|
+
|
|
273
|
+
## Results Summary
|
|
274
|
+
The Results Summary module extracts key results from HyPhy `.json` output files and saves them into a single Excel summary file.
|
|
275
|
+
|
|
276
|
+
When running HyPhy analyses for multiple genes, users often need to open many JSON files manually to check p-values, LRT scores, and significant branches. This module reduces that manual work by collecting the main results automatically.
|
|
277
|
+
|
|
278
|
+
**Key Features**
|
|
279
|
+
* **Drag and drop** HyPhy `.json` result files directly into the interface.
|
|
280
|
+
* **Detects the HyPhy model** used for each file automatically.
|
|
281
|
+
* **Extracts model-specific** summary results efficiently.
|
|
282
|
+
* **Saves all extracted results** into a single Excel workbook.
|
|
283
|
+
* **Organizes results** into separate sheets by HyPhy model.
|
|
284
|
+
* **Highlights significant results** automatically (e.g., p-value < 0.05).
|
|
285
|
+
* **Shows parsing errors** or skipped files clearly in the log console.
|
|
286
|
+
|
|
287
|
+
**Input**
|
|
288
|
+
Use the final `.json` output files generated by the HyPhy Execution module:
|
|
289
|
+
```text
|
|
290
|
+
GeneName_aln_rec_v1_MMDD_BUSTED.json
|
|
291
|
+
GeneName_aln_rec_v1_MMDD_RELAX.json
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
**Output**
|
|
295
|
+
The module generates a single organized Excel workbook:
|
|
296
|
+
```text
|
|
297
|
+
HyPhy_results_summary_MMDD.xlsx
|
|
298
|
+
```
|
|
299
|
+
|
|
300
|
+
---
|
|
301
|
+
|
|
302
|
+
## Supported HyPhy Models
|
|
303
|
+
Currently supported models across all modules include:
|
|
304
|
+
* BUSTED
|
|
305
|
+
* aBSREL
|
|
306
|
+
* RELAX
|
|
307
|
+
* FEL
|
|
308
|
+
* MEME
|
|
309
|
+
* FUBAR
|
|
310
|
+
* SLAC
|
|
311
|
+
|
|
312
|
+
---
|
|
313
|
+
|
|
314
|
+
## Dependencies
|
|
315
|
+
To use the full HYphlow pipeline, you can set up your environment using either Conda (recommended) or Pip.
|
|
316
|
+
|
|
317
|
+
**Option 1: Using Conda **
|
|
318
|
+
The easiest way to install all dependencies, including the HyPhy engine and Python packages, is to use the provided `environment.yml` file. This creates a dedicated virtual environment.
|
|
319
|
+
```bash
|
|
320
|
+
conda env create -f environment.yml
|
|
321
|
+
conda activate hyphlow_env
|
|
322
|
+
hyphy --version
|
|
323
|
+
```
|
|
324
|
+
|
|
325
|
+
**Option 2: Using Pip**
|
|
326
|
+
If you already have the HyPhy engine installed on your system and prefer not to use Conda, you can manually install the required Python GUI and bioinformatics packages using `requirements.txt`.
|
|
327
|
+
```bash
|
|
328
|
+
# Ensure HyPhy is installed first: conda install -c bioconda hyphy
|
|
329
|
+
pip install -r requirements.txt
|
|
330
|
+
```
|
|
File without changes
|