fetchm2 0.1.1__tar.gz → 0.1.2__tar.gz

{fetchm2-0.1.1/src/fetchm2.egg-info → fetchm2-0.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fetchm2
-Version: 0.1.1
+Version: 0.1.2
 Summary: Standalone comprehensive genome metadata standardization and sequence download toolkit.
 Author-email: Tasnimul Arabi Anik <arabianik987@gmail.com>
 License-Expression: MIT
@@ -40,6 +40,12 @@ FetchM2 is a comprehensive standalone command-line toolkit for bacterial genome
 
 FetchM2 is designed as the updated successor to the original FetchM standalone tool. It keeps the same practical command-line workflow, but adds many more standardized metadata fields, richer filtering, packaged curation rules, audit outputs, and reproducible test data.
 
+Recommended one-command workflow:
+
+```bash
+fetchm2 run --input ncbi_dataset.tsv --outdir results --download
+```
+
 ## Key Features
 
 - Standalone command-line tool installable with `pip` or a conda environment.
@@ -136,6 +142,29 @@ Run metadata standardization and sequence download in one command:
 fetchm2 run --input ncbi_dataset.tsv --outdir results --download
 ```
 
+## Typical Species/Genus Workflow
+
+1. Download an NCBI Genome Datasets TSV or CSV for your target species or genus.
+2. Run FetchM2:
+
+```bash
+fetchm2 run --input ncbi_dataset.tsv --outdir results --download
+```
+
+3. Review the main outputs:
+
+- `results/metadata_output/fetchm2_clean.csv`
+- `results/metadata_analysis/metadata_analysis_report.md`
+- `results/audit/standardization_audit.md`
+- `results/audit/production_readiness_gate.md`
+- `results/sequence/`
+
+For large NCBI retrieval jobs without an API key, use a conservative request delay:
+
+```bash
+fetchm2 run --input ncbi_dataset.tsv --outdir results --download --workers 3 --sleep 0.4
+```
+
 ## Metadata Retrieval Workflow
 
 FetchM2 can work in two modes.
@@ -175,6 +204,8 @@ fetchm2 metadata --help
 fetchm2 run --help
 fetchm2 seq --help
 fetchm2 audit --help
+fetchm2 validate --help
+fetchm2 analyze --help
 ```
 
 ### `fetchm2 metadata`
@@ -256,6 +287,16 @@ fetchm2 audit \
   --outdir results/audit_rerun
 ```
 
+### `fetchm2 validate`
+
+Runs the same production-readiness checks as `audit`, but names the workflow explicitly for CLI validation:
+
+```bash
+fetchm2 validate \
+  --input results/metadata_output/fetchm2_clean.csv \
+  --outdir results/validation
+```
+
 ### `fetchm2 analyze`
 
 Generates metadata analysis outputs from any existing clean metadata CSV.
@@ -283,6 +324,41 @@ FetchM2 writes:
 - `metadata_analysis/tables/numeric_summary.csv`
 - `metadata_analysis/figures/*.png`
 
+Typical output structure:
+
+```text
+results/
+├── metadata_output/
+│   ├── fetchm2_clean.csv
+│   ├── fetchm2_clean.tsv
+│   └── fetchm2_report.md
+├── metadata_analysis/
+│   ├── metadata_analysis_report.md
+│   ├── tables/
+│   └── figures/
+├── audit/
+│   ├── standardization_summary.csv
+│   ├── standardization_audit.md
+│   ├── production_readiness_gate.md
+│   ├── production_readiness_gate.json
+│   ├── top_host_review_needed.csv
+│   ├── non_country_values_in_country.csv
+│   ├── country_continent_mismatch.csv
+│   ├── country_subcontinent_mismatch.csv
+│   ├── invalid_collection_years.csv
+│   ├── invalid_host_like_sample_type.csv
+│   ├── source_like_mapped_hosts.csv
+│   ├── source_like_unmapped_hosts_for_review.csv
+│   ├── broad_vocabulary_leakage.csv
+│   ├── sequence_readiness.csv
+│   └── rule_count_summary.csv
+└── sequence/
+    ├── *.fna
+    ├── failed_accessions.txt
+    ├── sequence_download_summary.csv
+    └── fetchm2_sequence_cache.sqlite3
+```
+
 ## Standardized Metadata Fields
 
 FetchM2 keeps the original input columns and adds standardized fields.
@@ -304,6 +380,7 @@ Original FetchM had host-oriented metadata summaries. FetchM2 expands this into
 - `Host_Genus`
 - `Host_Species`
 - `Host_Common_Name`
+- `Host_Context_SD`
 - `Host_Match_Method`
 - `Host_Confidence`
 - `Host_Review_Status`
@@ -428,6 +505,7 @@ FetchM2 ships deterministic rules in `src/fetchm2/data/`:
 - `controlled_categories.csv`
 - `approved_broad_categories.csv`
 - `geography_reviewed_rules.csv`
+- `collection_date_reviewed_rules.csv`
 - `country_mapping.json`
 
 These rules let the standalone tool produce richer standardized fields without needing a web database.
@@ -453,6 +531,7 @@ python -m build
 python -m twine check dist/*
 python -m pip install dist/fetchm2-*.whl
 fetchm2 metadata --input examples/offline_metadata.tsv --outdir smoke_out --offline
+fetchm2 validate --input smoke_out/metadata_output/fetchm2_clean.csv --outdir smoke_out/validation
 fetchm2 seq --input smoke_out/metadata_output/fetchm2_clean.csv --outdir smoke_seq --country Bangladesh --check-only
 ```
 

{fetchm2-0.1.1 → fetchm2-0.1.2}/README.md

@@ -4,6 +4,12 @@ FetchM2 is a comprehensive standalone command-line toolkit for bacterial genome
 
 FetchM2 is designed as the updated successor to the original FetchM standalone tool. It keeps the same practical command-line workflow, but adds many more standardized metadata fields, richer filtering, packaged curation rules, audit outputs, and reproducible test data.
 
+Recommended one-command workflow:
+
+```bash
+fetchm2 run --input ncbi_dataset.tsv --outdir results --download
+```
+
 ## Key Features
 
 - Standalone command-line tool installable with `pip` or a conda environment.
@@ -100,6 +106,29 @@ Run metadata standardization and sequence download in one command:
 fetchm2 run --input ncbi_dataset.tsv --outdir results --download
 ```
 
+## Typical Species/Genus Workflow
+
+1. Download an NCBI Genome Datasets TSV or CSV for your target species or genus.
+2. Run FetchM2:
+
+```bash
+fetchm2 run --input ncbi_dataset.tsv --outdir results --download
+```
+
+3. Review the main outputs:
+
+- `results/metadata_output/fetchm2_clean.csv`
+- `results/metadata_analysis/metadata_analysis_report.md`
+- `results/audit/standardization_audit.md`
+- `results/audit/production_readiness_gate.md`
+- `results/sequence/`
+
+For large NCBI retrieval jobs without an API key, use a conservative request delay:
+
+```bash
+fetchm2 run --input ncbi_dataset.tsv --outdir results --download --workers 3 --sleep 0.4
+```
+
 ## Metadata Retrieval Workflow
 
 FetchM2 can work in two modes.
@@ -139,6 +168,8 @@ fetchm2 metadata --help
 fetchm2 run --help
 fetchm2 seq --help
 fetchm2 audit --help
+fetchm2 validate --help
+fetchm2 analyze --help
 ```
 
 ### `fetchm2 metadata`
@@ -220,6 +251,16 @@ fetchm2 audit \
   --outdir results/audit_rerun
 ```
 
+### `fetchm2 validate`
+
+Runs the same production-readiness checks as `audit`, but names the workflow explicitly for CLI validation:
+
+```bash
+fetchm2 validate \
+  --input results/metadata_output/fetchm2_clean.csv \
+  --outdir results/validation
+```
+
 ### `fetchm2 analyze`
 
 Generates metadata analysis outputs from any existing clean metadata CSV.
@@ -247,6 +288,41 @@ FetchM2 writes:
 - `metadata_analysis/tables/numeric_summary.csv`
 - `metadata_analysis/figures/*.png`
 
+Typical output structure:
+
+```text
+results/
+├── metadata_output/
+│   ├── fetchm2_clean.csv
+│   ├── fetchm2_clean.tsv
+│   └── fetchm2_report.md
+├── metadata_analysis/
+│   ├── metadata_analysis_report.md
+│   ├── tables/
+│   └── figures/
+├── audit/
+│   ├── standardization_summary.csv
+│   ├── standardization_audit.md
+│   ├── production_readiness_gate.md
+│   ├── production_readiness_gate.json
+│   ├── top_host_review_needed.csv
+│   ├── non_country_values_in_country.csv
+│   ├── country_continent_mismatch.csv
+│   ├── country_subcontinent_mismatch.csv
+│   ├── invalid_collection_years.csv
+│   ├── invalid_host_like_sample_type.csv
+│   ├── source_like_mapped_hosts.csv
+│   ├── source_like_unmapped_hosts_for_review.csv
+│   ├── broad_vocabulary_leakage.csv
+│   ├── sequence_readiness.csv
+│   └── rule_count_summary.csv
+└── sequence/
+    ├── *.fna
+    ├── failed_accessions.txt
+    ├── sequence_download_summary.csv
+    └── fetchm2_sequence_cache.sqlite3
+```
+
 ## Standardized Metadata Fields
 
 FetchM2 keeps the original input columns and adds standardized fields.
@@ -268,6 +344,7 @@ Original FetchM had host-oriented metadata summaries. FetchM2 expands this into
 - `Host_Genus`
 - `Host_Species`
 - `Host_Common_Name`
+- `Host_Context_SD`
 - `Host_Match_Method`
 - `Host_Confidence`
 - `Host_Review_Status`
@@ -392,6 +469,7 @@ FetchM2 ships deterministic rules in `src/fetchm2/data/`:
 - `controlled_categories.csv`
 - `approved_broad_categories.csv`
 - `geography_reviewed_rules.csv`
+- `collection_date_reviewed_rules.csv`
 - `country_mapping.json`
 
 These rules let the standalone tool produce richer standardized fields without needing a web database.
@@ -417,6 +495,7 @@ python -m build
 python -m twine check dist/*
 python -m pip install dist/fetchm2-*.whl
 fetchm2 metadata --input examples/offline_metadata.tsv --outdir smoke_out --offline
+fetchm2 validate --input smoke_out/metadata_output/fetchm2_clean.csv --outdir smoke_out/validation
 fetchm2 seq --input smoke_out/metadata_output/fetchm2_clean.csv --outdir smoke_seq --country Bangladesh --check-only
 ```
 

{fetchm2-0.1.1 → fetchm2-0.1.2}/docs/STANDARDIZATION.md

@@ -11,13 +11,14 @@ The packaged files live in `src/fetchm2/data/`:
 - `controlled_categories.csv`: source, sample, environment, disease, and health-state rules.
 - `approved_broad_categories.csv`: allowed broad-category vocabulary.
 - `geography_reviewed_rules.csv`: reviewed special geography cases.
+- `collection_date_reviewed_rules.csv`: reviewed date phrases that need explicit year recovery.
 - `country_mapping.json`: country, continent, and subcontinent mapping extracted from public FetchM.
 
 ## Output Fields
 
 FetchM2 writes the original input columns plus standardized columns including:
 
-- `Host_SD`, `Host_TaxID`, `Host_Rank`, host lineage fields, match method, confidence, and review status.
+- `Host_SD`, `Host_TaxID`, `Host_Rank`, host lineage fields, `Host_Context_SD`, match method, confidence, and review status.
 - `Sample_Type_SD`, `Isolation_Source_SD`, `Isolation_Site_SD`.
 - `Environment_Medium_SD`, `Environment_Broad_Scale_SD`, `Environment_Local_Scale_SD`.
 - `Host_Disease_SD`, `Host_Health_State_SD`.
@@ -28,7 +29,19 @@ FetchM2 writes the original input columns plus standardized columns including:
 The audit gate fails on obvious category leakage:
 
 - non-country values in `Country`
+- country-continent and country-subcontinent mismatches
+- invalid or future `Collection_Year` values
 - host-only values in `Sample_Type_SD`
 - unapproved `Isolation_Source_SD_Broad` values
+- missing required clean standardized columns
 
 Warnings are used for curation backlogs such as high host review counts.
+
+Each `metadata`, `audit`, or `validate` run writes:
+
+- `standardization_audit.md`
+- `standardization_summary.csv`
+- `production_readiness_gate.md`
+- `production_readiness_gate.json`
+- `top_host_review_needed.csv`
+- issue-specific review CSVs for country, year, sample/source/host leakage, broad vocabulary, and sequence readiness.

{fetchm2-0.1.1 → fetchm2-0.1.2}/docs/VALIDATION_REPORT.md

@@ -1,7 +1,7 @@
 # FetchM2 Validation Report
 
 Validation date: 2026-05-05
-Current validation target: `fetchm2 0.1.1`
+Current validation target: `fetchm2 0.1.2`
 
 ## Source Baselines
 
@@ -27,6 +27,7 @@ FetchM2 packages these deterministic rule resources:
 | `controlled_categories.csv` | 7,505 |
 | `approved_broad_categories.csv` | 50 |
 | `geography_reviewed_rules.csv` | 16 |
+| `collection_date_reviewed_rules.csv` | 0 reviewed rows |
 | `country_mapping.json` | 202 countries/regions |
 
 ## Commands Validated
@@ -37,6 +38,7 @@ The following commands were validated in isolated environments:
 fetchm2 --version
 fetchm2 metadata --input examples/offline_metadata.tsv --outdir /tmp/fetchm2_smoke --offline
 fetchm2 audit --input /tmp/fetchm2_smoke/metadata_output/fetchm2_clean.csv --outdir /tmp/fetchm2_smoke_audit
+fetchm2 validate --input /tmp/fetchm2_smoke/metadata_output/fetchm2_clean.csv --outdir /tmp/fetchm2_smoke_validation
 fetchm2 seq --input /tmp/fetchm2_smoke/metadata_output/fetchm2_clean.csv --outdir /tmp/fetchm2_smoke_seq --country Bangladesh --check-only
 ```
 
@@ -45,28 +47,30 @@ fetchm2 seq --input /tmp/fetchm2_smoke/metadata_output/fetchm2_clean.csv --outdi
 Regression tests:
 
 ```text
-7 passed
+11 passed
 ```
 
-Package build for `0.1.1`:
+Package build for `0.1.2`:
 
 ```text
 python -m build: passed
 twine check dist/*: passed
 ```
 
-Wheel installation for `0.1.1`:
+Wheel installation for `0.1.2`:
 
 ```text
-pip install fetchm2-0.1.1-py3-none-any.whl: passed
-fetchm2 --version: fetchm2 0.1.1
+pip install fetchm2-0.1.2-py3-none-any.whl: passed
+fetchm2 --version: fetchm2 0.1.2
 ```
 
-Wheel smoke test for `0.1.1`:
+Wheel smoke test for `0.1.2`:
 
 ```text
 metadata command: production gate PASS
 metadata analysis outputs: generated
+validate command: production gate PASS
+sequence check-only: selected 1 accession and completed without download
 ```
 
 Live NCBI smoke test:
@@ -159,7 +163,73 @@ FetchM2 `0.1.0` has been released on GitHub and PyPI.
 
 The `0.1.1` update has passed local tests, package build validation, clean wheel installation, offline packaged-test validation, 20-row live BioSample validation, and full `test.tsv` live BioSample validation.
 
-Known scope notes for `0.1.1`:
+## Additional 0.1.2 CLI Hardening Validation
+
+The 0.1.2 development update adds:
+
+- `fetchm2 validate`
+- `production_readiness_gate.md`
+- `production_readiness_gate.json`
+- issue-specific validation CSVs for country, collection year, sample/source/host leakage, broad vocabulary, missing columns, and sequence readiness
+- `Host_Context_SD`
+- packaged `collection_date_reviewed_rules.csv` support
+- final terminal summaries after `metadata`, `run`, `audit`, and `validate`
+- `sequence_download_summary.csv` in `--check-only` mode
+
+Local 0.1.2 validation:
+
+```text
+pytest: 11 passed
+examples/offline_metadata.tsv metadata run: production gate PASS
+validate command: production gate PASS
+sequence check-only: wrote failed_accessions.txt and sequence_download_summary.csv
+```
+
+Live 20-row `test.tsv` validation under 0.1.2:
+
+```text
+Rows scanned: 20
+Metadata fetch status: 20 ok
+Metadata fetch errors: 0
+Production gate: PASS
+Host TaxID mapped: 14 / 20 (70.0%)
+Country present: 14 / 20 (70.0%)
+Collection year present: 18 / 20 (90.0%)
+Invalid host-like Sample_Type_SD rows: 0
+Non-country values in Country rows: 0
+Country-continent mismatch rows: 0
+Country-subcontinent mismatch rows: 0
+Invalid/future Collection_Year rows: 0
+Unapproved broad-category rows: 0
+Metadata analysis figures generated: 47
+```
+
+Live full `test.tsv` validation under 0.1.2:
+
+```text
+Rows scanned: 200
+Production gate: PASS
+Host TaxID mapped: 92 / 200 (46.0%)
+Host review needed: 54
+Country present: 178 / 200 (89.0%)
+Collection year present: 196 / 200 (98.0%)
+Sample_Type_SD present: 66 / 200 (33.0%)
+Isolation_Source_SD present: 28 / 200 (14.0%)
+Isolation_Site_SD present: 46 / 200 (23.0%)
+Environment_Medium_SD present: 10 / 200 (5.0%)
+Invalid host-like Sample_Type_SD rows: 0
+Non-country values in Country rows: 0
+Country-continent mismatch rows: 0
+Country-subcontinent mismatch rows: 0
+Invalid/future Collection_Year rows: 0
+Unapproved Isolation_Source_SD_Broad rows: 0
+Unapproved broad-category rows: 0
+Source-like mapped host rows: 6
+Source-like unmapped host rows: 8
+Sequence-readiness issue rows: 0
+```
+
+Known scope notes for `0.1.2`:
 
 - host lineage is bundled for common hosts and optionally enriched with `taxonkit` when installed
 - embeddings/BGE are intentionally not used in production standardization

{fetchm2-0.1.1 → fetchm2-0.1.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "fetchm2"
-version = "0.1.1"
+version = "0.1.2"
 description = "Standalone comprehensive genome metadata standardization and sequence download toolkit."
 readme = "README.md"
 requires-python = ">=3.10"

{fetchm2-0.1.1 → fetchm2-0.1.2}/src/fetchm2/__init__.py

@@ -2,4 +2,4 @@
 
 from __future__ import annotations
 
-__version__ = "0.1.1"
+__version__ = "0.1.2"

{fetchm2-0.1.1 → fetchm2-0.1.2}/src/fetchm2/analysis.py

@@ -55,6 +55,7 @@ CORE_STANDARDIZED_FIELDS = [
     "Host_Genus",
     "Host_Species",
     "Host_Common_Name",
+    "Host_Context_SD",
     "Host_Match_Method",
     "Host_Confidence",
     "Host_Review_Status",