fetchm2 0.1.0__tar.gz

fetchm2-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Tasnimul Arabi Anik
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

fetchm2-0.1.0/MANIFEST.in

@@ -0,0 +1,5 @@
+include README.md
+include LICENSE
+recursive-include src/fetchm2/data *.csv *.json
+recursive-include examples *.tsv
+recursive-include docs *.md

fetchm2-0.1.0/PKG-INFO

@@ -0,0 +1,208 @@
+Metadata-Version: 2.4
+Name: fetchm2
+Version: 0.1.0
+Summary: Standalone comprehensive genome metadata standardization and sequence download toolkit.
+Author-email: Tasnimul Arabi Anik <arabianik987@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/Tasnimul-Arabi-Anik/FetchM2
+Project-URL: Repository, https://github.com/Tasnimul-Arabi-Anik/FetchM2
+Project-URL: Issues, https://github.com/Tasnimul-Arabi-Anik/FetchM2/issues
+Keywords: NCBI,BioSample,metadata,genomics,standardization,sequence-download
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pandas>=2.0
+Requires-Dist: requests>=2.31
+Requires-Dist: tqdm>=4.66
+Requires-Dist: matplotlib>=3.7
+Requires-Dist: seaborn>=0.13
+Requires-Dist: plotly>=5.20
+Requires-Dist: kaleido<1.0.0,>=0.2.1
+Requires-Dist: xmltodict>=0.13
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: build>=1.2; extra == "dev"
+Requires-Dist: twine>=5.0; extra == "dev"
+Dynamic: license-file
+
+# FetchM2
+
+FetchM2 is a standalone command-line toolkit for genome metadata retrieval, comprehensive metadata standardization, audit reporting, and optional sequence download.
+
+It keeps the simple standalone installation model of the original public [`FetchM`](https://github.com/Tasnimul-Arabi-Anik/FetchM), while packaging deterministic rule files and QA concepts developed in FetchM Web.
+
+## What FetchM2 Does
+
+- Reads NCBI Genome Datasets TSV/CSV exports.
+- Optionally fetches linked BioSample metadata from NCBI.
+- Standardizes host, country/geography, collection year, sample type, isolation source, isolation site, environment medium, host disease, and host health state.
+- Adds host TaxID, rank, lineage fields, match method, confidence, and review status.
+- Writes clean metadata tables and audit reports.
+- Downloads genome FASTA files from NCBI with flexible filters.
+- Runs offline on already annotated tables for reproducible tests and local standardization.
+
+## Installation
+
+Recommended clean environment:
+
+```bash
+python -m venv fetchm2-env
+source fetchm2-env/bin/activate
+pip install fetchm2
+```
+
+For development from source:
+
+```bash
+git clone https://github.com/Tasnimul-Arabi-Anik/FetchM2.git
+cd FetchM2
+python -m pip install -e ".[dev]"
+pytest
+```
+
+FetchM2 uses Python dependencies only. `taxonkit` is optional. If available, FetchM2 can use it to enrich less common host TaxIDs with lineage fields; common host lineages are bundled.
+
+## Quick Start
+
+Offline smoke test using the bundled example:
+
+```bash
+fetchm2 metadata --input examples/offline_metadata.tsv --outdir demo_out --offline
+```
+
+Full BioSample metadata retrieval:
+
+```bash
+fetchm2 metadata --input ncbi_dataset.tsv --outdir results
+```
+
+With NCBI API key:
+
+```bash
+export NCBI_API_KEY=YOUR_NCBI_API_KEY
+fetchm2 metadata --input ncbi_dataset.tsv --outdir results --workers 6 --sleep 0.15
+```
+
+All-in-one metadata plus sequence download:
+
+```bash
+fetchm2 run --input ncbi_dataset.tsv --outdir results --download
+```
+
+Filtered sequence download from a clean table:
+
+```bash
+fetchm2 seq \
+  --input results/metadata_output/fetchm2_clean.csv \
+  --outdir results/sequence \
+  --host "Homo sapiens" \
+  --country Bangladesh \
+  --year-from 2018 \
+  --year-to 2024
+```
+
+## Main Commands
+
+```bash
+fetchm2 metadata --help
+fetchm2 run --help
+fetchm2 seq --help
+fetchm2 audit --help
+```
+
+## Metadata Outputs
+
+FetchM2 writes:
+
+- `metadata_output/fetchm2_clean.csv`
+- `metadata_output/fetchm2_clean.tsv`
+- `metadata_output/fetchm2_report.md`
+- `audit/standardization_summary.csv`
+- `audit/top_host_review_needed.csv`
+- `audit/standardization_audit.md`
+
+Important standardized fields include:
+
+- `Host_SD`, `Host_TaxID`, `Host_Rank`, `Host_Superkingdom`, `Host_Phylum`, `Host_Class`, `Host_Order`, `Host_Family`, `Host_Genus`, `Host_Species`
+- `Host_Common_Name`, `Host_Match_Method`, `Host_Confidence`, `Host_Review_Status`
+- `Sample_Type_SD`, `Sample_Type_SD_Broad`
+- `Isolation_Source_SD`, `Isolation_Source_SD_Broad`
+- `Isolation_Site_SD`
+- `Environment_Medium_SD`, `Environment_Medium_SD_Broad`
+- `Environment_Broad_Scale_SD`, `Environment_Local_Scale_SD`
+- `Host_Disease_SD`, `Host_Health_State_SD`
+- `Country`, `Continent`, `Subcontinent`, `Collection_Year`
+
+## Sequence Download Options
+
+FetchM2 supports filtering by:
+
+- host
+- host rank
+- country
+- continent
+- subcontinent
+- sample type
+- isolation source
+- environment medium
+- collection year range
+- maximum genomes
+
+Use `--check-only` to audit a sequence output directory without downloading.
+
+## API Keys
+
+For NCBI, prefer environment variables:
+
+```bash
+export NCBI_API_KEY=YOUR_NCBI_API_KEY
+export NCBI_EMAIL=you@example.com
+```
+
+Do not place API keys in scripts, notebooks, README files, or Git commits.
+
+## Design Compared With FetchM and FetchM Web
+
+FetchM2 uses the original FetchM standalone flow as the command-line baseline:
+
+- metadata
+- run
+- seq
+- SQLite cache
+- NCBI BioSample fetch
+- sequence download from NCBI FTP
+
+FetchM2 adds FetchM Web-style standardized metadata fields and deterministic rule files:
+
+- host synonyms and negative host rules
+- controlled source/sample/environment categories
+- approved broad vocabulary
+- production-style audit gate
+- richer sequence filtering on standardized fields
+
+FetchM2 intentionally does not use embeddings or AI for production mappings. Embeddings can be used later as a review assistant, but final production rules should remain deterministic and auditable.
+
+## Testing
+
+Run:
+
+```bash
+pytest
+python -m build
+python -m pip install dist/fetchm2-*.whl
+fetchm2 metadata --input examples/offline_metadata.tsv --outdir smoke_out --offline
+fetchm2 seq --input smoke_out/metadata_output/fetchm2_clean.csv --outdir smoke_seq --country Bangladesh --check-only
+```
+
+## License
+
+MIT License.

fetchm2-0.1.0/README.md

@@ -0,0 +1,172 @@
+# FetchM2
+
+FetchM2 is a standalone command-line toolkit for genome metadata retrieval, comprehensive metadata standardization, audit reporting, and optional sequence download.
+
+It keeps the simple standalone installation model of the original public [`FetchM`](https://github.com/Tasnimul-Arabi-Anik/FetchM), while packaging deterministic rule files and QA concepts developed in FetchM Web.
+
+## What FetchM2 Does
+
+- Reads NCBI Genome Datasets TSV/CSV exports.
+- Optionally fetches linked BioSample metadata from NCBI.
+- Standardizes host, country/geography, collection year, sample type, isolation source, isolation site, environment medium, host disease, and host health state.
+- Adds host TaxID, rank, lineage fields, match method, confidence, and review status.
+- Writes clean metadata tables and audit reports.
+- Downloads genome FASTA files from NCBI with flexible filters.
+- Runs offline on already annotated tables for reproducible tests and local standardization.
+
+## Installation
+
+Recommended clean environment:
+
+```bash
+python -m venv fetchm2-env
+source fetchm2-env/bin/activate
+pip install fetchm2
+```
+
+For development from source:
+
+```bash
+git clone https://github.com/Tasnimul-Arabi-Anik/FetchM2.git
+cd FetchM2
+python -m pip install -e ".[dev]"
+pytest
+```
+
+FetchM2 uses Python dependencies only. `taxonkit` is optional. If available, FetchM2 can use it to enrich less common host TaxIDs with lineage fields; common host lineages are bundled.
+
+## Quick Start
+
+Offline smoke test using the bundled example:
+
+```bash
+fetchm2 metadata --input examples/offline_metadata.tsv --outdir demo_out --offline
+```
+
+Full BioSample metadata retrieval:
+
+```bash
+fetchm2 metadata --input ncbi_dataset.tsv --outdir results
+```
+
+With NCBI API key:
+
+```bash
+export NCBI_API_KEY=YOUR_NCBI_API_KEY
+fetchm2 metadata --input ncbi_dataset.tsv --outdir results --workers 6 --sleep 0.15
+```
+
+All-in-one metadata plus sequence download:
+
+```bash
+fetchm2 run --input ncbi_dataset.tsv --outdir results --download
+```
+
+Filtered sequence download from a clean table:
+
+```bash
+fetchm2 seq \
+  --input results/metadata_output/fetchm2_clean.csv \
+  --outdir results/sequence \
+  --host "Homo sapiens" \
+  --country Bangladesh \
+  --year-from 2018 \
+  --year-to 2024
+```
+
+## Main Commands
+
+```bash
+fetchm2 metadata --help
+fetchm2 run --help
+fetchm2 seq --help
+fetchm2 audit --help
+```
+
+## Metadata Outputs
+
+FetchM2 writes:
+
+- `metadata_output/fetchm2_clean.csv`
+- `metadata_output/fetchm2_clean.tsv`
+- `metadata_output/fetchm2_report.md`
+- `audit/standardization_summary.csv`
+- `audit/top_host_review_needed.csv`
+- `audit/standardization_audit.md`
+
+Important standardized fields include:
+
+- `Host_SD`, `Host_TaxID`, `Host_Rank`, `Host_Superkingdom`, `Host_Phylum`, `Host_Class`, `Host_Order`, `Host_Family`, `Host_Genus`, `Host_Species`
+- `Host_Common_Name`, `Host_Match_Method`, `Host_Confidence`, `Host_Review_Status`
+- `Sample_Type_SD`, `Sample_Type_SD_Broad`
+- `Isolation_Source_SD`, `Isolation_Source_SD_Broad`
+- `Isolation_Site_SD`
+- `Environment_Medium_SD`, `Environment_Medium_SD_Broad`
+- `Environment_Broad_Scale_SD`, `Environment_Local_Scale_SD`
+- `Host_Disease_SD`, `Host_Health_State_SD`
+- `Country`, `Continent`, `Subcontinent`, `Collection_Year`
+
+## Sequence Download Options
+
+FetchM2 supports filtering by:
+
+- host
+- host rank
+- country
+- continent
+- subcontinent
+- sample type
+- isolation source
+- environment medium
+- collection year range
+- maximum genomes
+
+Use `--check-only` to audit a sequence output directory without downloading.
+
+## API Keys
+
+For NCBI, prefer environment variables:
+
+```bash
+export NCBI_API_KEY=YOUR_NCBI_API_KEY
+export NCBI_EMAIL=you@example.com
+```
+
+Do not place API keys in scripts, notebooks, README files, or Git commits.
+
+## Design Compared With FetchM and FetchM Web
+
+FetchM2 uses the original FetchM standalone flow as the command-line baseline:
+
+- metadata
+- run
+- seq
+- SQLite cache
+- NCBI BioSample fetch
+- sequence download from NCBI FTP
+
+FetchM2 adds FetchM Web-style standardized metadata fields and deterministic rule files:
+
+- host synonyms and negative host rules
+- controlled source/sample/environment categories
+- approved broad vocabulary
+- production-style audit gate
+- richer sequence filtering on standardized fields
+
+FetchM2 intentionally does not use embeddings or AI for production mappings. Embeddings can be used later as a review assistant, but final production rules should remain deterministic and auditable.
+
+## Testing
+
+Run:
+
+```bash
+pytest
+python -m build
+python -m pip install dist/fetchm2-*.whl
+fetchm2 metadata --input examples/offline_metadata.tsv --outdir smoke_out --offline
+fetchm2 seq --input smoke_out/metadata_output/fetchm2_clean.csv --outdir smoke_seq --country Bangladesh --check-only
+```
+
+## License
+
+MIT License.

fetchm2-0.1.0/docs/RELEASE_CHECKLIST.md

@@ -0,0 +1,30 @@
+# FetchM2 Release Checklist
+
+Use this checklist before each GitHub/PyPI release.
+
+## Pre-release
+
+- Update `src/fetchm2/__init__.py` version.
+- Update `pyproject.toml` version.
+- Run `pytest`.
+- Run `python -m build`.
+- Run `python -m twine check dist/*`.
+- Install the wheel in a clean environment.
+- Run offline metadata smoke test.
+- Run sequence `--check-only` smoke test.
+- Run a small live NCBI smoke test when network access is available.
+- Confirm no API keys, tokens, caches, or output directories are committed.
+
+## Release
+
+- Commit all source, data, docs, tests, and examples.
+- Tag the commit, for example `v0.1.0`.
+- Push branch and tag to GitHub.
+- Upload `dist/*` to PyPI using an environment variable or secure prompt.
+
+## Post-release
+
+- Install from PyPI in a fresh environment.
+- Run `fetchm2 --version`.
+- Run the offline smoke test from the PyPI-installed package.
+- Record results in `docs/VALIDATION_REPORT.md`.

fetchm2-0.1.0/docs/SEQUENCE_DOWNLOAD.md

@@ -0,0 +1,45 @@
+# FetchM2 Sequence Download
+
+FetchM2 downloads genome FASTA files from the public NCBI genomes FTP layout using the assembly accession and assembly name in `fetchm2_clean.csv`.
+
+## Typical Workflow
+
+```bash
+fetchm2 metadata --input ncbi_dataset.tsv --outdir results
+fetchm2 seq --input results/metadata_output/fetchm2_clean.csv --outdir results/sequence
+```
+
+## Filtered Download
+
+You can filter using standardized metadata:
+
+```bash
+fetchm2 seq \
+  --input results/metadata_output/fetchm2_clean.csv \
+  --outdir results/sequence_human_bd \
+  --host "Homo sapiens" \
+  --country Bangladesh \
+  --year-from 2018 \
+  --year-to 2024
+```
+
+Supported filters include:
+
+- `--host`
+- `--host-rank`
+- `--country`
+- `--continent`
+- `--subcontinent`
+- `--sample-type`
+- `--isolation-source`
+- `--environment-medium`
+- `--year-from` and `--year-to`
+- `--max-genomes`
+
+## Check Only
+
+Use `--check-only` to compare expected accessions against an output directory without downloading:
+
+```bash
+fetchm2 seq --input fetchm2_clean.csv --outdir sequence --check-only
+```

fetchm2-0.1.0/docs/STANDARDIZATION.md

@@ -0,0 +1,34 @@
+# FetchM2 Standardization Notes
+
+FetchM2 packages deterministic standardization rules from FetchM Web so the standalone tool can reproduce the major standardized metadata fields without requiring a web database.
+
+## Rule Sources
+
+The packaged files live in `src/fetchm2/data/`:
+
+- `host_synonyms.csv`: exact and broad host mappings with TaxID.
+- `host_negative_rules.csv`: values blocked from `Host_SD`, including source/material/lab artifacts.
+- `controlled_categories.csv`: source, sample, environment, disease, and health-state rules.
+- `approved_broad_categories.csv`: allowed broad-category vocabulary.
+- `geography_reviewed_rules.csv`: reviewed special geography cases.
+- `country_mapping.json`: country, continent, and subcontinent mapping extracted from public FetchM.
+
+## Output Fields
+
+FetchM2 writes the original input columns plus standardized columns including:
+
+- `Host_SD`, `Host_TaxID`, `Host_Rank`, host lineage fields, match method, confidence, and review status.
+- `Sample_Type_SD`, `Isolation_Source_SD`, `Isolation_Site_SD`.
+- `Environment_Medium_SD`, `Environment_Broad_Scale_SD`, `Environment_Local_Scale_SD`.
+- `Host_Disease_SD`, `Host_Health_State_SD`.
+- `Country`, `Continent`, `Subcontinent`, `Collection_Year`.
+
+## Production Gate
+
+The audit gate fails on obvious category leakage:
+
+- non-country values in `Country`
+- host-only values in `Sample_Type_SD`
+- unapproved `Isolation_Source_SD_Broad` values
+
+Warnings are used for curation backlogs such as high host review counts.

fetchm2-0.1.0/docs/VALIDATION_REPORT.md

@@ -0,0 +1,99 @@
+# FetchM2 Validation Report
+
+Validation date: 2026-05-05
+
+## Source Baselines
+
+FetchM2 was built using:
+
+- public standalone FetchM GitHub repository as the CLI/workflow baseline
+- FetchM Web standardization rule files as deterministic packaged data
+
+The public FetchM baseline inspected from GitHub was commit:
+
+```text
+11070b9 Avoid Chrome requirement for map export
+```
+
+## Packaged Rule Counts
+
+FetchM2 packages these deterministic rule resources:
+
+| Rule file | Rows |
+| --- | ---: |
+| `host_synonyms.csv` | 7,113 |
+| `host_negative_rules.csv` | 408 |
+| `controlled_categories.csv` | 7,505 |
+| `approved_broad_categories.csv` | 50 |
+| `geography_reviewed_rules.csv` | 16 |
+| `country_mapping.json` | 202 countries/regions |
+
+## Commands Validated
+
+The following commands were validated in isolated environments:
+
+```bash
+fetchm2 --version
+fetchm2 metadata --input examples/offline_metadata.tsv --outdir /tmp/fetchm2_smoke --offline
+fetchm2 audit --input /tmp/fetchm2_smoke/metadata_output/fetchm2_clean.csv --outdir /tmp/fetchm2_smoke_audit
+fetchm2 seq --input /tmp/fetchm2_smoke/metadata_output/fetchm2_clean.csv --outdir /tmp/fetchm2_smoke_seq --country Bangladesh --check-only
+```
+
+## Test Results
+
+Regression tests:
+
+```text
+6 passed
+```
+
+Package build:
+
+```text
+python -m build: passed
+twine check dist/*: passed
+```
+
+Wheel installation:
+
+```text
+pip install dist/fetchm2-0.1.0-py3-none-any.whl: passed
+```
+
+Wheel smoke test:
+
+```text
+metadata command: production gate PASS
+sequence check-only: selected 1 accession, wrote failed_accessions.txt as expected
+```
+
+Live NCBI smoke test:
+
+```text
+Input: first two rows from public FetchM test.tsv
+BioSample records requested: 1 unique BioSample
+Result: metadata command completed and production gate PASS
+```
+
+## Regression Scenarios Covered
+
+The test suite checks:
+
+- `human blood` maps to `Host_SD=Homo sapiens`, `Host_TaxID=9606`, and `Sample_Type_SD=blood`
+- `bacteria culture` is blocked from `Host_SD`
+- `Hospital` is blocked from standardized `Country`
+- `turkey breast sandwich` does not become a country or host
+- `water deer` remains a valid host and is not treated as water
+- metadata CLI writes clean outputs and audit files
+- sequence `--check-only` works with standardized filters
+
+## Release Readiness
+
+FetchM2 is ready for an initial `0.1.0` GitHub/PyPI release as a standalone alpha package.
+
+Known scope limits for `0.1.0`:
+
+- host lineage is bundled for common hosts and optionally enriched with `taxonkit` when installed
+- figures are not yet as extensive as FetchM Web dashboards
+- embeddings/BGE are intentionally not used in production standardization
+- large-scale sequence download was not run during this validation to avoid unnecessary NCBI load

fetchm2-0.1.0/examples/offline_metadata.tsv

@@ -0,0 +1,6 @@
+Assembly Accession	Assembly Name	Organism Name	ANI Check status	CheckM completeness	Host	Collection Date	Geographic Location	Isolation Source	Sample Type	Isolation Site	Environment Medium	Host Disease	Host Health State
+GCF_000001.1	Test assembly 1	Escherichia coli	OK	99.1	human blood	2020-04-01	Bangladesh: Dhaka	blood	blood	vein		healthy	healthy
+GCF_000002.1	Test assembly 2	Salmonella enterica	OK	98.4	cattle feces	2019	United States: Texas	feces	feces	rectum		diseased	mastitis
+GCF_000003.1	Test assembly 3	Vibrio cholerae	OK	97.0	not collected	2021-07	soil around the Arctic Ocean	soil	soil		soil		
+GCF_000004.1	Test assembly 4	Staphylococcus aureus	Failed	96.2	bacteria culture	2022	Hospital	pure culture	culture			unknown	
+GCF_000005.1	Test assembly 5	Campylobacter jejuni	OK	95.5	turkey breast sandwich	2023	ground turkey	food	meat		retail food