adaptiseq 0.1.3__tar.gz

adaptiseq-0.1.3/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 swargo98
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

adaptiseq-0.1.3/MANIFEST.in

@@ -0,0 +1,10 @@
+include LICENSE
+include README.md
+include pyproject.toml
+
+# Ship the full test tree (fixtures, conftest, local servers) so the sdist is
+# self-testable, not the half tree distutils' legacy test*.py auto-include leaves.
+graft tests
+
+# Never ship compiled artifacts.
+global-exclude *.py[cod]

adaptiseq-0.1.3/PKG-INFO

@@ -0,0 +1,400 @@
+Metadata-Version: 2.4
+Name: adaptiseq
+Version: 0.1.3
+Summary: Fast, importable downloader for public sequencing data (GSA/SRA/ENA/DDBJ/GEO) with a segmented, resumable, self-tuning batch engine.
+Author: swargo98
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/swargo98/adaptiSeq
+Project-URL: Documentation, https://swargo98.github.io/adaptiSeq/
+Project-URL: Repository, https://github.com/swargo98/adaptiSeq
+Project-URL: Issues, https://github.com/swargo98/adaptiSeq/issues
+Keywords: bioinformatics,sequencing,SRA,ENA,GSA,DDBJ,GEO,download,fastq,aspera
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Operating System :: MacOS
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: aiohttp>=3.8
+Requires-Dist: aioftp>=0.21
+Requires-Dist: numpy>=1.21
+Provides-Extra: test
+Requires-Dist: pytest>=7.0; extra == "test"
+Provides-Extra: docs
+Requires-Dist: mkdocs-material>=9.5; extra == "docs"
+Dynamic: license-file
+
+<h1 align="center">
+  <img src="https://raw.githubusercontent.com/swargo98/adaptiSeq/main/adaptiseq/assets/logo.png" alt="adaptiSeq" width="480">
+</h1>
+
+<p align="center"><b>An adaptive tool to fetch public sequencing data</b></p>
+
+<p align="center">
+  <a href="https://github.com/swargo98/adaptiSeq/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT"></a>
+  <a href="https://www.python.org/"><img src="https://img.shields.io/badge/python-3.10%2B-blue.svg" alt="Python 3.10+"></a>
+  <a href="https://pypi.org/project/adaptiseq/"><img src="https://img.shields.io/pypi/v/adaptiseq.svg" alt="PyPI version"></a>
+  <a href="https://github.com/swargo98/adaptiSeq/actions/workflows/ci.yml"><img src="https://github.com/swargo98/adaptiSeq/actions/workflows/ci.yml/badge.svg" alt="CI status"></a>
+  <a href="https://swargo98.github.io/adaptiSeq/"><img src="https://img.shields.io/badge/docs-online-0a6b6b" alt="Documentation"></a>
+</p>
+
+<p align="center">
+  <a href="https://swargo98.github.io/adaptiSeq/"><b>📖 Documentation</b></a>
+  &nbsp;·&nbsp;
+  <a href="#installation">Installation</a>
+  &nbsp;·&nbsp;
+  <a href="#usage">Usage</a>
+  &nbsp;·&nbsp;
+  <a href="#related-tools">Comparison</a>
+</p>
+
+<!-- PyPI/bioconda badges resolve once the package is published; until then see Installation. -->
+
+## Description
+
+**adaptiSeq** is a fast, importable Python tool that downloads sequencing data and
+metadata from **[GSA](https://ngdc.cncb.ac.cn/gsa/)**,
+**[SRA](https://www.ncbi.nlm.nih.gov/sra/)**,
+**[ENA](https://www.ebi.ac.uk/ena/)**,
+**[DDBJ](https://www.ddbj.nig.ac.jp/)**, and
+**[GEO](https://www.ncbi.nlm.nih.gov/geo/)**. It is built for the workload real
+pipelines actually have: **lists of accessions**, downloaded in parallel, with a
+**segmented, resumable, self-tuning** engine and a **real Python API**.
+
+> **Important**
+> To use adaptiSeq your system must be **connected to the network** and able to
+> reach the databases over **HTTP, HTTPS, and (optionally) FTP / Aspera**.
+
+## Update Notes
+
+### 0.1.3
+- **Fix**: a segmented download whose `.part` file is missing at finalize (e.g.
+  every segment's connection was refused) now fails cleanly and retries, instead
+  of raising a confusing `FileNotFoundError` from an unguarded `.part` rename.
+
+### 0.1.2
+- **Fix**: the notebook event-loop handling from 0.1.1 now also works when
+  `nest_asyncio` is active (it previously hung/errored on batch downloads). The
+  API detects a re-entrant loop and drives the coroutine on it directly; plain
+  Jupyter still uses a worker thread. Works with or without `nest_asyncio`.
+
+### 0.1.1
+- **Fix**: the Python API (`fetch`/`resolve`) now works inside a running asyncio
+  event loop (Jupyter / Google Colab / IPython), where it previously raised
+  `asyncio.run() cannot be called from a running event loop`. The CLI was
+  unaffected.
+
+### 0.1.0 (initial release)
+- First public release: segmented engine + adaptive batch pool + parallel
+  metadata resolution + adaptive Aspera + Python API.
+- Validated against the **real ENA Aspera** endpoint with a genuine IBM `ascp`.
+- Offline test suite (fixtures, engine, batch, adaptive) green; CI on Python
+  3.10–3.12.
+
+<details>
+<summary>Design highlights</summary>
+
+- **Segmented, resumable HTTP(S)/FTP engine** (the default) with `.part`/`.part.meta`
+  resume — interrupt and re-run to continue, not restart.
+- **Batch-parallel download** of accession lists through an asyncio worker pool.
+- **Adaptive concurrency**: an adaptive controller (top-down knee search by
+  default) tunes how many downloads run at once from measured throughput.
+- **Parallel metadata resolution** under polite per-endpoint rate limits.
+- **Adaptive parallel Aspera** via an efficiency-hysteresis controller.
+- **Python API** (`fetch`, `resolve`, `get_metadata`) that returns values and
+  raises typed exceptions.
+- Correctly downloads **3-file runs** (orphan/barcode + `_1` + `_2`) that iSeq
+  mishandles.
+
+</details>
+
+## Features
+- **Multiple database support**: GSA / SRA / ENA / DDBJ / GEO.
+- **Multiple input formats**: Project, Study, BioSample, Sample, Experiment, or
+  Run accession — single, or a file of many (mixed databases allowed).
+- **Metadata download**: per-accession sample metadata (`-m`).
+<details>
+<summary>More features</summary>
+
+- **File-format selection**: direct gzip FASTQ (`-g`) or SRA→FASTQ conversion (`-q`).
+- **Multi-threaded conversion/compression** (`-t`).
+- **File merging** per Experiment/Sample/Study (`-e`).
+- **Batch-parallel download** of accession lists (`-j`), with **adaptive**
+  concurrency (`--adaptive`) and **parallel metadata resolution** (`--meta-jobs`).
+- **Segmented, resumable transfers** with HTTPS-first transport selection.
+- **Aspera high-speed download** for GSA/ENA (`-a`), adaptive for ENA.
+- **Automatic retry** (up to three rounds) and **md5 / `vdb-validate` verification**.
+- **Importable Python API** with typed exceptions and `py.typed`.
+- **Error handling**: clear, actionable messages with suggested solutions.
+
+</details>
+
+## Installation
+
+### 1. From PyPI (once published)
+```bash
+pip install adaptiseq            # not yet on PyPI — see "From source" below
+```
+
+### 2. From source
+```bash
+git clone https://github.com/swargo98/adaptiSeq.git
+cd adaptiSeq
+pip install -e .
+adaptiseq --version
+```
+
+### 3. Conda environment
+```bash
+conda env create -f environment.yml && conda activate adaptiseq
+```
+
+External tools: `wget` (always), `sra-tools`, `pigz`, `md5sum`,
+`axel` (classic `-p`), and IBM `ascp` (`-a`). adaptiSeq requires each **only when a
+run actually needs it**. Full detail: **[docs/installation.md](docs/installation.md)**.
+
+## Example
+
+1. Download all Runs and metadata for an accession.
+```bash
+adaptiseq -i PRJNA211801
+```
+
+2. Batch download a list, directly as gzip FASTQ, through the adaptive pool.
+```bash
+adaptiseq -i SRR_Acc_List.txt -g
+```
+
+See **[docs/examples.md](docs/examples.md)** for more.
+
+## Usage
+
+```
+$ adaptiseq --help
+
+Usage:
+  adaptiseq -i accession [options]
+
+Required option:
+  -i, --input     [text|file]   Single accession or a file containing multiple accessions.
+                                Note: only one accession per line in the file.
+
+Optional options:
+  -m, --metadata                Skip the sequencing data downloads and only fetch metadata.
+  -g, --gzip                    Download FASTQ files in gzip format directly (*.fastq.gz).
+  -q, --fastq                   Convert SRA files to FASTQ format.
+  -t, --threads   int           Threads for SRA->FASTQ / compression (default: 8).
+  -e, --merge     [ex|sa|st]    Merge fastq files per Experiment, Sample, or Study.
+  -d, --database  [ena|sra]     Force database for SRA data (default: auto-detect).
+  -p, --parallel  int           axel connection count (classic engine only), e.g. -p 10.
+  -a, --aspera                  Use Aspera (ascp); GSA/ENA only.
+  -s, --speed     int           Download speed limit (MB/s) (default: 1000).
+  -k, --skip-md5                Skip the md5 check for downloaded files.
+  -r, --protocol  [ftp|https]   ENA transport (default: auto, HTTPS-first).
+  -Q, --quiet                   Suppress download progress bars.
+  -o, --output    text          Output directory (created if missing; default: cwd).
+  --engine        [segmented|classic]   Download engine (default: segmented).
+  --segment-size  int           Target segment size in MB (default: 512).
+  --max-segments  int           Max connections per file (default: 8).
+  --max-conns-per-host int      Global cap on connections to one host (default: 8).
+  -j, --jobs      int           Max worker-pool size for batch download (default: 20).
+  --adaptive / --no-adaptive    Gradient adaptive-concurrency controller (default: on).
+  --probe-window  int           Adaptive probe window in seconds (default: 5).
+  --cc-penalty    float         Worker-cost penalty K (default: 1.01).
+  --meta-jobs     int           Parallelism for metadata/URL resolution (default: 3).
+  --aspera-efficiency float     Keep an extra ascp worker only above this efficiency (0.70).
+  -h, --help                    Show the help information.
+  -v, --version                 Show the program version.
+```
+
+### 1. `-i`, `--input`
+
+Input the accession to download, or a **file** with one accession per line
+(databases may be mixed). adaptiSeq retrieves the accession's metadata, then
+downloads each Run it contains.
+
+```bash
+adaptiseq -i PRJNA211801
+adaptiseq -i accessions.txt        # batch
+```
+
+Currently **supports 6 accession formats** across **5 databases**:
+
+| Databases | BioProject | Study | BioSample | Sample | Experiment | Run  |
+| --------- | ---------- | ----- | --------- | ------ | ---------- | ---- |
+| **GSA**   | PRJC       | CRA   | SAMC      | \      | CRX        | CRR  |
+| **SRA**   | PRJNA      | SRP   | SAMN      | SRS    | SRX        | SRR  |
+| **ENA**   | PRJEB      | ERP   | SAME      | ERS    | ERX        | ERR  |
+| **DDBJ**  | PRJDB      | DRP   | SAMD      | DRS    | DRX        | DRR  |
+| **GEO**   | GSE        | \     | GSM       | \      | \          | \    |
+
+GEO `GSE/GSM` accessions are resolved to their associated `PRJNA/SAMN` and then
+downloaded from SRA. Every contained Run is **md5-checked**; on mismatch adaptiSeq
+retries up to **three rounds**, recording results in `success.log` / `fail.log`.
+
+### 2. `-m`, `--metadata`
+
+Download only the sample metadata and skip the sequence data.
+
+```bash
+adaptiseq -i PRJNA211801 -m
+adaptiseq -i CRR343031 -m
+```
+
+ENA → TSV (`${accession}.metadata.tsv`); GSA → CSV + project XLSX
+(`${accession}.metadata.csv`, `CRA*.metadata.xlsx`). Details:
+**[docs/usage/metadata.md](docs/usage/metadata.md)**.
+
+### 3. `-g`, `--gzip`
+
+Download FASTQ directly in gzip format; if unavailable on ENA, download the `.sra`
+and convert with `fasterq-dump` + `pigz`.
+
+```bash
+adaptiseq -i SRR1178105 -g
+```
+
+### 4. `-q`, `--fastq`
+
+Convert the downloaded `.sra` into FASTQ (single-cell friendly: `I1/R1/R2/R3`).
+
+```bash
+adaptiseq -i SRR1178105 -q -t 10
+```
+
+### 5. `-e`, `--merge`
+
+Merge a group of Runs into one FASTQ per Experiment (`ex`), Sample (`sa`), or
+Study (`st`).
+
+```bash
+adaptiseq -i SRX003906 -g -e ex
+```
+
+### 6. `-d` / `-r` / `-p` / `-s` / `-k` / `-Q`
+
+Force database (`-d ena|sra`), ENA transport (`-r https|ftp`, default auto
+HTTPS-first), classic-engine axel connections (`-p`, with `--engine classic`),
+speed cap (`-s` MB/s), skip md5 (`-k`), and quiet (`-Q`). See
+**[docs/usage/download.md](docs/usage/download.md)**.
+
+### 7. Batch & adaptive (`-j`, `--adaptive`, `--meta-jobs`)
+
+For accession lists, adaptiSeq resolves in parallel (`--meta-jobs`) then downloads
+through an asyncio worker pool (`-j`) whose active size the adaptive controller
+tunes (`--adaptive`; `topdown` by default, `ASEQ_ADAPTIVE_MODE=bottomup` for the
+gradient controller — see `docs/ADAPTIVE_CONTROLLER.md`).
+
+```bash
+adaptiseq -i accessions.txt -g -j 8
+adaptiseq -i accessions.txt -g --no-adaptive
+```
+
+Full detail: **[docs/usage/batch.md](docs/usage/batch.md)**.
+
+### 8. `-a`, `--aspera`
+
+Use IBM Aspera (`ascp`); **GSA/ENA only**. ENA uses an **adaptive** parallel pool
+(efficiency hysteresis); GSA is sequential, best-effort (Huawei-Cloud preferred).
+
+```bash
+adaptiseq -i PRJNA211801 -a -g
+```
+
+Detail: **[docs/usage/aspera.md](docs/usage/aspera.md)**.
+
+### 9. Python API
+
+```python
+from adaptiseq import fetch, resolve, get_metadata
+
+rows   = get_metadata("SRR7706354")               # parsed metadata rows
+urls   = resolve("SRR7706354", database="ena")    # resolved URLs (no download)
+result = fetch("accessions.txt", outdir="data/", gzip=True, jobs=20, adaptive=True)
+print(result.success_ids, result.fail_ids, result.failed)
+```
+
+The API never calls `sys.exit` and never prints colour codes; it raises the typed
+exceptions in `adaptiseq.errors`. Full reference:
+**[docs/usage/python-api.md](docs/usage/python-api.md)**.
+
+## Output
+
+- **SRA / ENA / DDBJ / GEO** accessions:
+
+| Output        | Description |
+| ------------- | ----------- |
+| SRA files     | Convert to FASTQ with `-q` |
+| `.metadata.tsv` | Metadata for the query accession |
+| `success.log` | Runs downloaded successfully |
+| `fail.log`    | Runs that failed |
+
+- **GSA** accessions:
+
+| Output         | Description |
+| -------------- | ----------- |
+| GSA files      | Mostly `*.gz` (a few bam/tar/bz2) |
+| `.metadata.csv`  | Metadata for the query accession |
+| `.metadata.xlsx` | Project metadata (xlsx) for the accession |
+| `success.log`  | Runs downloaded successfully |
+| `fail.log`     | Runs that failed |
+
+## Documentation
+
+Full documentation lives in **[`docs/`](docs/README.md)**:
+[Installation](docs/installation.md) ·
+[Usage overview](docs/usage/README.md) ·
+[Downloading](docs/usage/download.md) ·
+[Metadata](docs/usage/metadata.md) ·
+[Merging](docs/usage/merge.md) ·
+[Batch & adaptive](docs/usage/batch.md) ·
+[Aspera](docs/usage/aspera.md) ·
+[Python API](docs/usage/python-api.md) ·
+[Method details](docs/methods.md) ·
+[Examples](docs/examples.md) ·
+[FAQ](docs/faq.md).
+
+## Related tools
+
+adaptiSeq sits among other downloaders for public sequencing data —
+[iSeq](https://github.com/BioOmics/iSeq),
+[fastq-dl](https://github.com/rpetit3/fastq-dl),
+[fetchngs](https://github.com/nf-core/fetchngs),
+[pysradb](https://github.com/saketkc/pysradb), and
+[Kingfisher](https://github.com/wwood/kingfisher-download). A comparison:
+
+| Software | Languages | Databases | Accessions | Formats | Methods | Metadata | MD5 | Resumable | Parallel | Merge | Skip done | Conda |
+| -------- | --------- | --------- | ---------- | ------- | ------- | :------: | :-: | :-------: | :------: | :---: | :-------: | :---: |
+| **adaptiSeq** | Python | GSA, SRA, ENA, DDBJ, GEO | All | fq, fq.gz, sra, bam | segmented http(s)/ftp, aspera, wget/axel | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | TBD (pip ✔) |
+| [iSeq](https://github.com/BioOmics/iSeq) | Shell | GSA, SRA, ENA, DDBJ, GEO | All | fq, fq.gz, sra, bam | wget, axel, aspera | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
+| [SRA Toolkit](https://github.com/ncbi/sra-tools) | C | SRA, ENA, DDBJ | All except Run | fq, fq.gz, sra | prefetch | ❌ | ✔ | ✔ | ❌ | ❌ | ✔ | ✔ |
+| [enaBrowserTools](https://github.com/enasequence/enaBrowserTools) | Python | SRA, ENA, DDBJ | All except GSA/GEO | fq, fq.gz, sra | urllib, aspera | ✔ | ✔ | ✔ | ❌ | ❌ | ✔ | ✔ |
+| [fastq-dl](https://github.com/rpetit3/fastq-dl) | Python | SRA, ENA, DDBJ | All except GSA/GEO | fq, fq.gz, sra | wget | ✔ | ✔ | ❌ | ❌ | ✔ | ✔ | ✔ |
+| [fetchngs](https://github.com/nf-core/fetchngs) | Python | SRA, ENA, DDBJ, GEO | All except GSA | fq, fq.gz | wget, aspera, prefetch | ✔ | ✔ | ✔ | ❌ | ❌ | ✔ | ❌ |
+| [pysradb](https://github.com/saketkc/pysradb) | Python | SRA, ENA, DDBJ, GEO | All except GSA | fq, fq.gz, sra, bam | requests, aspera | ✔ | ✔ | ✔ | ❌ | ❌ | ✔ | ✔ |
+| [Kingfisher](https://github.com/wwood/kingfisher-download) | Python | SRA, ENA, DDBJ | All except GSA/GEO | fq, fq.gz, sra | curl, aria2c, aspera | ✔ | ✔ | ❌ | ✔ | ❌ | ✔ | ✔ |
+
+Beyond this matrix, adaptiSeq adds what the single-shot downloaders do not: an
+**adaptive, batch-parallel** download path, **parallel metadata resolution**, a
+**segmented resumable** engine, and an importable **Python API**.
+
+## Contributing
+
+Contributions are welcome — please open an issue for bugs/suggestions, or fork,
+branch, and submit a pull request. The offline test suite runs with
+`python -m pytest`.
+
+## Cite us
+
+TBD (manuscript in preparation).
+
+## License
+
+This project is licensed under the [MIT License](LICENSE).