giftag 1.0.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- giftag-1.0.0/.gitignore +10 -0
- giftag-1.0.0/CHANGELOG.md +122 -0
- giftag-1.0.0/LICENSE +21 -0
- giftag-1.0.0/PKG-INFO +74 -0
- giftag-1.0.0/README.md +50 -0
- giftag-1.0.0/pyproject.toml +56 -0
- giftag-1.0.0/src/giftag/__init__.py +11 -0
- giftag-1.0.0/src/giftag/__main__.py +3 -0
- giftag-1.0.0/src/giftag/annotate.py +163 -0
- giftag-1.0.0/src/giftag/build.py +610 -0
- giftag-1.0.0/src/giftag/cli.py +363 -0
- giftag-1.0.0/src/giftag/data/dbcan_sub_db_v5-2-9_5-5-2026.tsv +503 -0
- giftag-1.0.0/src/giftag/database.py +107 -0
- giftag-1.0.0/src/giftag/fetch.py +218 -0
- giftag-1.0.0/src/giftag/genes.py +111 -0
- giftag-1.0.0/src/giftag/markers.py +89 -0
- giftag-1.0.0/src/giftag/search.py +152 -0
- giftag-1.0.0/src/giftag/sources.py +86 -0
- giftag-1.0.0/src/giftag/ui.py +222 -0
- giftag-1.0.0/tests/conftest.py +156 -0
- giftag-1.0.0/tests/test_giftag.py +290 -0
giftag-1.0.0/.gitignore
ADDED
|
@@ -0,0 +1,122 @@
|
|
|
1
|
+
# giftag changelog
|
|
2
|
+
|
|
3
|
+
Code, command-line and database-build changes. Newest first. Timestamps are
|
|
4
|
+
UTC.
|
|
5
|
+
|
|
6
|
+
giftag has two versions. The **package version** below follows this file. The
|
|
7
|
+
**database format** (`DB_FORMAT` in `giftag/__init__.py`, currently 1) changes
|
|
8
|
+
only when a directory written by `giftag build` can no longer be read by
|
|
9
|
+
`giftag annotate`; a format change is called out in its entry.
|
|
10
|
+
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
## 1.0.0 — 2026-10-05T04:15Z
|
|
14
|
+
|
|
15
|
+
### First release on PyPI
|
|
16
|
+
|
|
17
|
+
**What changed.** giftag is published on PyPI, so `pip install giftag` works. A GitHub Actions workflow builds the distributions, runs the tests against the built wheel, and uploads to PyPI through trusted publishing when a GitHub release is published. The source distribution now holds only the package, its offline tests, the README, the licence and this changelog; the benchmarks and the documentation site stay in the repository. The "under development" badge and note are replaced by a PyPI version badge.
|
|
18
|
+
|
|
19
|
+
**Why.** The quickstart has told users to run `pip install giftag` since the documentation site went up, but the package had never been uploaded. The source distribution was also shipping about 470 KB of benchmark results and documentation sources that an installation never uses.
|
|
20
|
+
|
|
21
|
+
**Effect.** Packaging and release process only. Annotation, search rules, output files and the database format (still 1) are unchanged from 0.2.0. The version is 1.0.0 because the command line and output files are now treated as stable.
|
|
22
|
+
|
|
23
|
+
### A dedicated documentation site connects giftag to gifter
|
|
24
|
+
|
|
25
|
+
**What changed.** A Quarto site now documents the build and annotation commands, output schema, marker-search coverage, source rules, validation limits, and the direct handoff to gifter. It uses gifter's palette, typography and logo, with navigation back to gifter. GitHub Actions renders it on pull requests and deploys it from `main`. The README is now a short entry point to those guides.
|
|
26
|
+
|
|
27
|
+
**Why.** giftag's CLI and coverage ledger need their own reference, while the interpretation of a GIFT call belongs in gifter's documentation.
|
|
28
|
+
|
|
29
|
+
**Effect.** Documentation and deployment workflow only. Annotation, search rules and output files are unchanged.
|
|
30
|
+
|
|
31
|
+
## 0.2.0 — 2026-10-04T05:05Z
|
|
32
|
+
|
|
33
|
+
### dbCAN-sub is searched on every protein
|
|
34
|
+
|
|
35
|
+
**What changed.** `annotate` now searches the dbCAN-sub clusters on every
|
|
36
|
+
protein. The earlier behaviour, searching a family's clusters only on proteins
|
|
37
|
+
with a domain of that family, is the opt-in `--gate-subfamilies`.
|
|
38
|
+
|
|
39
|
+
**Why.** The gate was meant to keep a subfamily call from standing without its
|
|
40
|
+
family. Measured against run_dbcan on *Bacteroides thetaiotaomicron* VPI-5482
|
|
41
|
+
it dropped 74 of 157 subfamily calls. Most of that was a bug: the gate looked
|
|
42
|
+
for a family hit named exactly `GH43` and ignored hits to dbCAN's official
|
|
43
|
+
subfamily models such as `GH43_18`. With that fixed it still dropped 13, and
|
|
44
|
+
gifter's subfamily evidence was curated against what run_dbcan emits.
|
|
45
|
+
|
|
46
|
+
**Effect.** Subfamily calls match run_dbcan on the kept clusters, 157 of 157.
|
|
47
|
+
The dbCAN step takes about twice as long; a genome takes about 45 s on 8 cores.
|
|
48
|
+
With `--gate-subfamilies` the result is 144 of 157.
|
|
49
|
+
|
|
50
|
+
### A hit to an official subfamily model counts as its family
|
|
51
|
+
|
|
52
|
+
**What changed.** When a dbCAN family-library hit is to a subfamily model
|
|
53
|
+
(`GH43_18`), giftag also emits the parent family marker (`GH43`). The row keeps
|
|
54
|
+
the subfamily model in its `profile` column.
|
|
55
|
+
|
|
56
|
+
**Why.** run_dbcan reports only the model that wins a region, so a GH43 protein
|
|
57
|
+
whose best hit is `GH43_18` carries no `GH43` accession. Fourteen of gifter's 53
|
|
58
|
+
CAZy family markers are in families with subfamily models, and those markers
|
|
59
|
+
missed every such protein. The narrower hit licenses the broader marker, never
|
|
60
|
+
the reverse.
|
|
61
|
+
|
|
62
|
+
**Effect.** This is the one place giftag's markers differ from run_dbcan's
|
|
63
|
+
literal output. On *B. thetaiotaomicron* it raises family marker rows from 104
|
|
64
|
+
to 191.
|
|
65
|
+
|
|
66
|
+
### dbCAN-sub is downloaded by byte range
|
|
67
|
+
|
|
68
|
+
**What changed.** `build` locates the blocks of the CAZy families gifter uses in
|
|
69
|
+
`dbCAN_sub.hmm` with range requests and downloads only those. A table of
|
|
70
|
+
profile counts per family ships for the pinned release
|
|
71
|
+
(`giftag/data/dbcan_sub_db_v5-2-9_5-5-2026.tsv`); every fetched block is checked
|
|
72
|
+
against it, and it supplies run_dbcan's Z. On a mismatch, or for a release
|
|
73
|
+
without a table, the whole library is streamed as before.
|
|
74
|
+
|
|
75
|
+
**Why.** The library is 5.1 GB and gifter needs the clusters of 47 of its 500
|
|
76
|
+
families.
|
|
77
|
+
|
|
78
|
+
**Effect.** 1.25 GB is downloaded instead of 5.1 GB, and the kept profiles are
|
|
79
|
+
byte-identical to the streamed result. A full build downloads about 2.9 GB.
|
|
80
|
+
|
|
81
|
+
### KOfam is fetched over FTP first
|
|
82
|
+
|
|
83
|
+
**What changed.** KOfam is downloaded from GenomeNet's FTP server, with HTTPS as
|
|
84
|
+
the fallback.
|
|
85
|
+
|
|
86
|
+
**Why.** GenomeNet throttles each HTTPS connection to tens of KB/s, which made
|
|
87
|
+
the 1.5 GB archive a day-long download. FTP is about thirty times faster.
|
|
88
|
+
|
|
89
|
+
**Effect.** KOfam takes about 25 minutes on an ordinary connection.
|
|
90
|
+
|
|
91
|
+
### Command line on Typer and Rich
|
|
92
|
+
|
|
93
|
+
**What changed.** The command line is rebuilt on Typer and Rich: grouped option
|
|
94
|
+
panels, live download and genome progress bars, elapsed-time log lines, and a
|
|
95
|
+
summary table after `build`, `annotate` and `info`. New global options
|
|
96
|
+
`--verbose`, `--quiet` and `--log-file PATH` go before the command. `-i`
|
|
97
|
+
accepts directories. A `version` command is added.
|
|
98
|
+
|
|
99
|
+
**Why.** A build downloads for the better part of an hour and an annotation run
|
|
100
|
+
may cover thousands of genomes; both need visible progress and a log that can
|
|
101
|
+
be kept.
|
|
102
|
+
|
|
103
|
+
**Effect.** Off a terminal the bars become plain periodic lines. `annotate()`
|
|
104
|
+
now returns a summary dictionary instead of a row count. `typer` and `rich` are
|
|
105
|
+
new dependencies. `--dbcan-dir` accepts `dbCAN-sub.hmm` as well as
|
|
106
|
+
`dbCAN_sub.hmm`. `annotate` writes its tables genome by genome, so an
|
|
107
|
+
interrupted run keeps what it finished; `giftag_run.json` marks a complete run.
|
|
108
|
+
|
|
109
|
+
## 0.1.0 — 2026-10-04T03:13Z
|
|
110
|
+
|
|
111
|
+
**What changed.** First version. `giftag build` compiles a profile database
|
|
112
|
+
from the KOfam, NCBIfam, Pfam and dbCAN releases gifter is curated against,
|
|
113
|
+
keeping the profiles gifter's markers need and the dbCAN profiles they compete
|
|
114
|
+
with. `giftag annotate` calls genes with pyrodigal, applies each source's own
|
|
115
|
+
acceptance rule, and writes the `genome_id`, `gene_id`, `namespace`,
|
|
116
|
+
`accession` table gifter reads.
|
|
117
|
+
|
|
118
|
+
**Why.** gifter's markers span four profile collections, and gifter cannot tell
|
|
119
|
+
an unsearched marker from an absent gene.
|
|
120
|
+
|
|
121
|
+
**Effect.** Database format 1. Every gifter marker gets a recorded status, and
|
|
122
|
+
those that cannot be searched are reported.
|
giftag-1.0.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
Scope: this license covers the original giftag software code and documentation.
|
|
2
|
+
It does not relicense any profile database giftag downloads; each source keeps
|
|
3
|
+
its own terms (see the README).
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
MIT No Attribution
|
|
7
|
+
|
|
8
|
+
Copyright 2026 Antton Alberdi
|
|
9
|
+
|
|
10
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy of
|
|
11
|
+
this software and associated documentation files (the "Software"), to deal in
|
|
12
|
+
the Software without restriction, including without limitation the rights to
|
|
13
|
+
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
|
|
14
|
+
the Software, and to permit persons to whom the Software is furnished to do so.
|
|
15
|
+
|
|
16
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
17
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
|
18
|
+
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
|
19
|
+
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
|
|
20
|
+
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
|
|
21
|
+
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
giftag-1.0.0/PKG-INFO
ADDED
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: giftag
|
|
3
|
+
Version: 1.0.0
|
|
4
|
+
Summary: Annotate genomes with exactly the markers gifter evaluates.
|
|
5
|
+
Project-URL: Homepage, https://github.com/alberdilab/giftag
|
|
6
|
+
Project-URL: Documentation, https://alberdilab.github.io/giftag/
|
|
7
|
+
Project-URL: Changelog, https://github.com/alberdilab/giftag/blob/main/CHANGELOG.md
|
|
8
|
+
Project-URL: gifter, https://github.com/alberdilab/gifter
|
|
9
|
+
Author: Antton Alberdi
|
|
10
|
+
License-Expression: MIT-0
|
|
11
|
+
License-File: LICENSE
|
|
12
|
+
Classifier: Development Status :: 5 - Production/Stable
|
|
13
|
+
Classifier: Operating System :: OS Independent
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
|
|
16
|
+
Requires-Python: >=3.9
|
|
17
|
+
Requires-Dist: pyhmmer>=0.12
|
|
18
|
+
Requires-Dist: pyrodigal>=3.5
|
|
19
|
+
Requires-Dist: rich>=13
|
|
20
|
+
Requires-Dist: typer<1,>=0.12
|
|
21
|
+
Provides-Extra: test
|
|
22
|
+
Requires-Dist: pytest>=7; extra == 'test'
|
|
23
|
+
Description-Content-Type: text/markdown
|
|
24
|
+
|
|
25
|
+
# giftag
|
|
26
|
+
|
|
27
|
+
[](https://github.com/alberdilab/giftag/actions/workflows/docs.yml)
|
|
28
|
+
[](https://alberdilab.github.io/giftag/)
|
|
29
|
+
[](https://pypi.org/project/giftag/)
|
|
30
|
+
[](LICENSE)
|
|
31
|
+
|
|
32
|
+
giftag annotates genome and protein FASTA files with the markers [gifter](https://alberdilab.github.io/gifter/) evaluates. It writes the gene-by-marker table gifter reads, together with search evidence and a record of which markers could be searched.
|
|
33
|
+
|
|
34
|
+
**[Read the giftag documentation](https://alberdilab.github.io/giftag/)** for the [quickstart](https://alberdilab.github.io/giftag/get-started.html), [commands](https://alberdilab.github.io/giftag/commands.html), [output files](https://alberdilab.github.io/giftag/output.html), [marker coverage](https://alberdilab.github.io/giftag/coverage.html), [methods and validation](https://alberdilab.github.io/giftag/methods.html), and the [handoff to gifter](https://alberdilab.github.io/giftag/with-gifter.html).
|
|
35
|
+
|
|
36
|
+
## Quickstart
|
|
37
|
+
|
|
38
|
+
Python 3.9 or newer is required. The full database build downloads about 2.9 GB from KOfam, NCBIfam, Pfam and dbCAN onto your machine and uses about 1.5 GB of disk space. Read the [source terms](https://alberdilab.github.io/giftag/coverage.html#source-terms) before building.
|
|
39
|
+
|
|
40
|
+
```sh
|
|
41
|
+
pip install giftag
|
|
42
|
+
giftag build
|
|
43
|
+
giftag annotate -i genomes/ -o annotations/
|
|
44
|
+
giftag info
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
Then use the marker table in R:
|
|
48
|
+
|
|
49
|
+
```r
|
|
50
|
+
markers <- read.delim("annotations/giftag_markers.tsv")
|
|
51
|
+
calls <- gifter::evaluate_gifts_community(markers)
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
Each FASTA file is one genome. `giftag_markers.tsv` contains `genome_id`, `gene_id`, `namespace` and `accession` for gifter, plus the profile, rule, score and threshold behind each accepted hit. The built database's `markers.tsv` identifies markers giftag could not search. Keep it and `giftag_run.json` with an analysis: an unsearched marker otherwise looks absent to gifter.
|
|
55
|
+
|
|
56
|
+
The tools make different claims. giftag observes marker evidence; gifter evaluates whether those observations complete a curated genomic capability. Neither tool infers expression, activity or phenotype from a hit.
|
|
57
|
+
|
|
58
|
+
## Development
|
|
59
|
+
|
|
60
|
+
```sh
|
|
61
|
+
pip install -e '.[test]'
|
|
62
|
+
pytest
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Documentation source is in [`docs/`](docs/). From the repository root, run `quarto render docs` to build the site locally. The GitHub Pages workflow renders it on pull requests and deploys it from `main`.
|
|
66
|
+
|
|
67
|
+
The [publication benchmark](benchmarks/RESULTS.md) compares giftag with
|
|
68
|
+
KofamScan, run_dbcan, direct HMMER and two InterProScan releases on a pinned
|
|
69
|
+
eight-genome panel. Its [reproduction guide](benchmarks/README.md) includes
|
|
70
|
+
the Mjolnir workflow, source hashes, environments and analysis scripts.
|
|
71
|
+
|
|
72
|
+
## Licensing
|
|
73
|
+
|
|
74
|
+
giftag's code and original documentation are MIT-0. giftag redistributes no profile libraries. `giftag build` downloads them from their publishers, whose terms apply; see [source terms](https://alberdilab.github.io/giftag/coverage.html#source-terms).
|
giftag-1.0.0/README.md
ADDED
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
# giftag
|
|
2
|
+
|
|
3
|
+
[](https://github.com/alberdilab/giftag/actions/workflows/docs.yml)
|
|
4
|
+
[](https://alberdilab.github.io/giftag/)
|
|
5
|
+
[](https://pypi.org/project/giftag/)
|
|
6
|
+
[](LICENSE)
|
|
7
|
+
|
|
8
|
+
giftag annotates genome and protein FASTA files with the markers [gifter](https://alberdilab.github.io/gifter/) evaluates. It writes the gene-by-marker table gifter reads, together with search evidence and a record of which markers could be searched.
|
|
9
|
+
|
|
10
|
+
**[Read the giftag documentation](https://alberdilab.github.io/giftag/)** for the [quickstart](https://alberdilab.github.io/giftag/get-started.html), [commands](https://alberdilab.github.io/giftag/commands.html), [output files](https://alberdilab.github.io/giftag/output.html), [marker coverage](https://alberdilab.github.io/giftag/coverage.html), [methods and validation](https://alberdilab.github.io/giftag/methods.html), and the [handoff to gifter](https://alberdilab.github.io/giftag/with-gifter.html).
|
|
11
|
+
|
|
12
|
+
## Quickstart
|
|
13
|
+
|
|
14
|
+
Python 3.9 or newer is required. The full database build downloads about 2.9 GB from KOfam, NCBIfam, Pfam and dbCAN onto your machine and uses about 1.5 GB of disk space. Read the [source terms](https://alberdilab.github.io/giftag/coverage.html#source-terms) before building.
|
|
15
|
+
|
|
16
|
+
```sh
|
|
17
|
+
pip install giftag
|
|
18
|
+
giftag build
|
|
19
|
+
giftag annotate -i genomes/ -o annotations/
|
|
20
|
+
giftag info
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
Then use the marker table in R:
|
|
24
|
+
|
|
25
|
+
```r
|
|
26
|
+
markers <- read.delim("annotations/giftag_markers.tsv")
|
|
27
|
+
calls <- gifter::evaluate_gifts_community(markers)
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
Each FASTA file is one genome. `giftag_markers.tsv` contains `genome_id`, `gene_id`, `namespace` and `accession` for gifter, plus the profile, rule, score and threshold behind each accepted hit. The built database's `markers.tsv` identifies markers giftag could not search. Keep it and `giftag_run.json` with an analysis: an unsearched marker otherwise looks absent to gifter.
|
|
31
|
+
|
|
32
|
+
The tools make different claims. giftag observes marker evidence; gifter evaluates whether those observations complete a curated genomic capability. Neither tool infers expression, activity or phenotype from a hit.
|
|
33
|
+
|
|
34
|
+
## Development
|
|
35
|
+
|
|
36
|
+
```sh
|
|
37
|
+
pip install -e '.[test]'
|
|
38
|
+
pytest
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Documentation source is in [`docs/`](docs/). From the repository root, run `quarto render docs` to build the site locally. The GitHub Pages workflow renders it on pull requests and deploys it from `main`.
|
|
42
|
+
|
|
43
|
+
The [publication benchmark](benchmarks/RESULTS.md) compares giftag with
|
|
44
|
+
KofamScan, run_dbcan, direct HMMER and two InterProScan releases on a pinned
|
|
45
|
+
eight-genome panel. Its [reproduction guide](benchmarks/README.md) includes
|
|
46
|
+
the Mjolnir workflow, source hashes, environments and analysis scripts.
|
|
47
|
+
|
|
48
|
+
## Licensing
|
|
49
|
+
|
|
50
|
+
giftag's code and original documentation are MIT-0. giftag redistributes no profile libraries. `giftag build` downloads them from their publishers, whose terms apply; see [source terms](https://alberdilab.github.io/giftag/coverage.html#source-terms).
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["hatchling>=1.18"]
|
|
3
|
+
build-backend = "hatchling.build"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
name = "giftag"
|
|
7
|
+
dynamic = ["version"]
|
|
8
|
+
description = "Annotate genomes with exactly the markers gifter evaluates."
|
|
9
|
+
readme = "README.md"
|
|
10
|
+
license = "MIT-0"
|
|
11
|
+
license-files = ["LICENSE"]
|
|
12
|
+
authors = [{ name = "Antton Alberdi" }]
|
|
13
|
+
requires-python = ">=3.9"
|
|
14
|
+
dependencies = [
|
|
15
|
+
"pyhmmer>=0.12",
|
|
16
|
+
"pyrodigal>=3.5",
|
|
17
|
+
"rich>=13",
|
|
18
|
+
"typer>=0.12,<1",
|
|
19
|
+
]
|
|
20
|
+
classifiers = [
|
|
21
|
+
"Development Status :: 5 - Production/Stable",
|
|
22
|
+
"Programming Language :: Python :: 3",
|
|
23
|
+
"Topic :: Scientific/Engineering :: Bio-Informatics",
|
|
24
|
+
"Operating System :: OS Independent",
|
|
25
|
+
]
|
|
26
|
+
|
|
27
|
+
[project.optional-dependencies]
|
|
28
|
+
test = ["pytest>=7"]
|
|
29
|
+
|
|
30
|
+
[project.urls]
|
|
31
|
+
Homepage = "https://github.com/alberdilab/giftag"
|
|
32
|
+
Documentation = "https://alberdilab.github.io/giftag/"
|
|
33
|
+
Changelog = "https://github.com/alberdilab/giftag/blob/main/CHANGELOG.md"
|
|
34
|
+
gifter = "https://github.com/alberdilab/gifter"
|
|
35
|
+
|
|
36
|
+
[project.scripts]
|
|
37
|
+
giftag = "giftag.cli:main"
|
|
38
|
+
|
|
39
|
+
[tool.hatch.version]
|
|
40
|
+
path = "src/giftag/__init__.py"
|
|
41
|
+
|
|
42
|
+
[tool.hatch.build.targets.wheel]
|
|
43
|
+
packages = ["src/giftag"]
|
|
44
|
+
|
|
45
|
+
# The source distribution carries the package and its offline tests. The
|
|
46
|
+
# benchmarks and the documentation site stay in the repository.
|
|
47
|
+
[tool.hatch.build.targets.sdist]
|
|
48
|
+
include = [
|
|
49
|
+
"/src/giftag",
|
|
50
|
+
"/tests/conftest.py",
|
|
51
|
+
"/tests/test_giftag.py",
|
|
52
|
+
"/CHANGELOG.md",
|
|
53
|
+
]
|
|
54
|
+
|
|
55
|
+
[tool.pytest.ini_options]
|
|
56
|
+
testpaths = ["tests"]
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
"""giftag: annotate genomes with exactly the markers gifter evaluates."""
|
|
2
|
+
|
|
3
|
+
__version__ = "1.0.0"
|
|
4
|
+
|
|
5
|
+
# The layout of a database directory written by `giftag build`. `annotate`
|
|
6
|
+
# refuses a directory whose format it does not know.
|
|
7
|
+
DB_FORMAT = 1
|
|
8
|
+
|
|
9
|
+
|
|
10
|
+
class GiftagError(Exception):
|
|
11
|
+
"""An error the command line reports as a message rather than a traceback."""
|
|
@@ -0,0 +1,163 @@
|
|
|
1
|
+
"""`giftag annotate`: genomes or proteins in, a gifter marker table out."""
|
|
2
|
+
|
|
3
|
+
import csv
|
|
4
|
+
import json
|
|
5
|
+
import time
|
|
6
|
+
from collections import Counter
|
|
7
|
+
from datetime import datetime, timezone
|
|
8
|
+
from pathlib import Path
|
|
9
|
+
|
|
10
|
+
from rich.markup import escape
|
|
11
|
+
|
|
12
|
+
from giftag import GiftagError, __version__, ui
|
|
13
|
+
from giftag.build import cazy_family
|
|
14
|
+
from giftag.database import Database
|
|
15
|
+
from giftag.genes import call_genes, genome_id, looks_nucleotide, read_fasta, write_fasta
|
|
16
|
+
|
|
17
|
+
MARKER_COLUMNS = (
|
|
18
|
+
"genome_id", "gene_id", "namespace", "accession", "source", "profile", "rule",
|
|
19
|
+
"score", "evalue", "threshold", "coverage", "target_from", "target_to",
|
|
20
|
+
)
|
|
21
|
+
GENOME_COLUMNS = ("genome_id", "input", "input_type", "gene_calling", "sequences",
|
|
22
|
+
"length_bp", "proteins", "marker_genes", "markers")
|
|
23
|
+
|
|
24
|
+
|
|
25
|
+
def annotate(inputs, outdir, db_dir, threads=0, mode="auto", input_type="auto",
|
|
26
|
+
gate_subfamilies=False):
|
|
27
|
+
started = time.monotonic()
|
|
28
|
+
database = Database(db_dir)
|
|
29
|
+
inputs = [Path(p) for p in inputs]
|
|
30
|
+
if not inputs:
|
|
31
|
+
raise GiftagError("no input files")
|
|
32
|
+
ids = [genome_id(p) for p in inputs]
|
|
33
|
+
duplicated = sorted(gid for gid, n in Counter(ids).items() if n > 1)
|
|
34
|
+
if duplicated:
|
|
35
|
+
raise GiftagError(f"inputs share genome IDs: {', '.join(duplicated)}")
|
|
36
|
+
outdir = Path(outdir)
|
|
37
|
+
outdir.mkdir(parents=True, exist_ok=True)
|
|
38
|
+
|
|
39
|
+
unsearchable = database.unsearchable
|
|
40
|
+
if unsearchable:
|
|
41
|
+
ui.warning(f"{len(unsearchable)} gifter markers cannot be searched with this database "
|
|
42
|
+
f"and will read as absent in gifter (see markers.tsv in the database)")
|
|
43
|
+
with ui.Task("loading profiles", done="profiles loaded in {elapsed}"):
|
|
44
|
+
database.load_all()
|
|
45
|
+
|
|
46
|
+
# Tables are written genome by genome, so a long run that stops part way
|
|
47
|
+
# keeps what it finished. giftag_run.json is written last and marks a run
|
|
48
|
+
# as complete.
|
|
49
|
+
(outdir / "giftag_run.json").unlink(missing_ok=True)
|
|
50
|
+
n_rows = 0
|
|
51
|
+
summary = []
|
|
52
|
+
width = len(str(len(inputs)))
|
|
53
|
+
with _Table(outdir / "giftag_markers.tsv", MARKER_COLUMNS) as markers, \
|
|
54
|
+
_Table(outdir / "giftag_genomes.tsv", GENOME_COLUMNS) as genomes, \
|
|
55
|
+
ui.Task("annotating", total=len(inputs), steps=False) as task:
|
|
56
|
+
for index, (path, gid) in enumerate(zip(inputs, ids), start=1):
|
|
57
|
+
began = time.monotonic()
|
|
58
|
+
name = escape(gid)
|
|
59
|
+
|
|
60
|
+
def stage(step):
|
|
61
|
+
task.update(description=f"{name} · {step}")
|
|
62
|
+
|
|
63
|
+
stage("reading")
|
|
64
|
+
records = read_fasta(path)
|
|
65
|
+
kind = input_type
|
|
66
|
+
if kind == "auto":
|
|
67
|
+
kind = "nucleotide" if looks_nucleotide(records) else "protein"
|
|
68
|
+
if kind == "nucleotide":
|
|
69
|
+
stage("calling genes")
|
|
70
|
+
proteins, used = call_genes(records, mode)
|
|
71
|
+
(outdir / "proteins").mkdir(exist_ok=True)
|
|
72
|
+
write_fasta(outdir / "proteins" / f"{gid}.faa", proteins)
|
|
73
|
+
calling = f"pyrodigal {used}"
|
|
74
|
+
else:
|
|
75
|
+
proteins, calling = records, "none (protein input)"
|
|
76
|
+
|
|
77
|
+
calls = database.search(proteins, cpus=threads, gate=gate_subfamilies, stage=stage)
|
|
78
|
+
rows = _label(gid, calls, database.labels)
|
|
79
|
+
markers.write(rows)
|
|
80
|
+
n_rows += len(rows)
|
|
81
|
+
row = dict(
|
|
82
|
+
genome_id=gid, input=str(path), input_type=kind, gene_calling=calling,
|
|
83
|
+
sequences=len(records),
|
|
84
|
+
length_bp=sum(len(s) for _, s in records) if kind == "nucleotide" else "",
|
|
85
|
+
proteins=len(proteins), marker_genes=len({r["gene_id"] for r in rows}),
|
|
86
|
+
markers=len({(r["namespace"], r["accession"]) for r in rows}),
|
|
87
|
+
)
|
|
88
|
+
genomes.write([row])
|
|
89
|
+
summary.append(row)
|
|
90
|
+
task.advance()
|
|
91
|
+
ui.info(f"[dim]{index:>{width}}/{len(inputs)}[/dim] [bold]{name}[/bold] · "
|
|
92
|
+
f"{len(proteins):,} proteins · {row['markers']:,} markers · "
|
|
93
|
+
f"{ui.format_duration(time.monotonic() - began)}")
|
|
94
|
+
|
|
95
|
+
manifest = database.manifest
|
|
96
|
+
run = {
|
|
97
|
+
"giftag_version": __version__,
|
|
98
|
+
"finished_utc": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
|
|
99
|
+
"seconds": round(time.monotonic() - started, 1),
|
|
100
|
+
"database": {
|
|
101
|
+
"path": str(database.path.resolve()),
|
|
102
|
+
"built_utc": manifest["built_utc"],
|
|
103
|
+
"gifter": manifest["gifter"],
|
|
104
|
+
"sources": {name: {k: v for k, v in meta.items() if k != "terms"}
|
|
105
|
+
for name, meta in manifest["sources"].items()},
|
|
106
|
+
"marker_status": manifest["marker_status"],
|
|
107
|
+
},
|
|
108
|
+
"parameters": {"mode": mode, "input_type": input_type, "threads": threads,
|
|
109
|
+
"gate_subfamilies": gate_subfamilies},
|
|
110
|
+
"genomes": len(inputs),
|
|
111
|
+
"rows": n_rows,
|
|
112
|
+
}
|
|
113
|
+
with open(outdir / "giftag_run.json", "w") as handle:
|
|
114
|
+
json.dump(run, handle, indent=2)
|
|
115
|
+
handle.write("\n")
|
|
116
|
+
return {"outdir": outdir, "rows": n_rows, "genomes": summary,
|
|
117
|
+
"seconds": run["seconds"], "unsearchable": len(unsearchable)}
|
|
118
|
+
|
|
119
|
+
|
|
120
|
+
def _label(gid, calls, labels):
|
|
121
|
+
"""Translate accepted calls into gifter's namespace/accession, one row per
|
|
122
|
+
gene and marker, keeping the strongest supporting hit."""
|
|
123
|
+
best = {}
|
|
124
|
+
for call in calls:
|
|
125
|
+
found = list(labels.get((call.source, call.profile), ()))
|
|
126
|
+
if call.source == "dbcan" and cazy_family(call.profile) != call.profile:
|
|
127
|
+
# A hit to an official subfamily model (GH43_18) is a hit to its
|
|
128
|
+
# family: the narrower evidence licenses the broader marker. The
|
|
129
|
+
# row keeps the subfamily model as its `profile`.
|
|
130
|
+
found += labels.get(("dbcan", cazy_family(call.profile)), ())
|
|
131
|
+
for namespace, accession in found:
|
|
132
|
+
key = (call.gene_id, namespace, accession)
|
|
133
|
+
if key not in best or call.score > best[key].score:
|
|
134
|
+
best[key] = call
|
|
135
|
+
rows = []
|
|
136
|
+
for (gene, namespace, accession), call in best.items():
|
|
137
|
+
rows.append(dict(
|
|
138
|
+
genome_id=gid, gene_id=gene, namespace=namespace, accession=accession,
|
|
139
|
+
source=call.source, profile=call.profile, rule=call.rule,
|
|
140
|
+
score=f"{call.score:.1f}", evalue=f"{call.evalue:.3g}",
|
|
141
|
+
threshold=f"{call.threshold:g}",
|
|
142
|
+
coverage="" if call.coverage is None else f"{call.coverage:.3f}",
|
|
143
|
+
target_from=call.target_from or "", target_to=call.target_to or "",
|
|
144
|
+
))
|
|
145
|
+
rows.sort(key=lambda r: (r["gene_id"], r["namespace"], r["accession"]))
|
|
146
|
+
return rows
|
|
147
|
+
|
|
148
|
+
|
|
149
|
+
class _Table:
|
|
150
|
+
def __init__(self, path, columns):
|
|
151
|
+
self._handle = open(path, "w", newline="")
|
|
152
|
+
self._writer = csv.DictWriter(self._handle, columns, delimiter="\t", lineterminator="\n")
|
|
153
|
+
self._writer.writeheader()
|
|
154
|
+
|
|
155
|
+
def write(self, rows):
|
|
156
|
+
self._writer.writerows(rows)
|
|
157
|
+
self._handle.flush()
|
|
158
|
+
|
|
159
|
+
def __enter__(self):
|
|
160
|
+
return self
|
|
161
|
+
|
|
162
|
+
def __exit__(self, *exc):
|
|
163
|
+
self._handle.close()
|