gapit 0.5.6__tar.gz → 0.6.2__tar.gz

{gapit-0.5.6 → gapit-0.6.2}/.github/workflows/ci.yml

@@ -4,6 +4,9 @@ on:
   push:
     branches: [main]
   pull_request:
+  schedule:
+    # Weekly clean-environment canary: no caches, locked dependencies.
+    - cron: "0 3 * * 1"
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
@@ -16,17 +19,32 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        env: [default, py311, py313]
+        env: [default]
     steps:
       - uses: actions/checkout@v4
       - uses: prefix-dev/setup-pixi@v0.8.1
         with:
           cache: true
       - run: pixi run -e ${{ matrix.env }} lint
-      - run: pixi run -e ${{ matrix.env }} fmt -- --check
+      - run: pixi run -e ${{ matrix.env }} fmt-check
       - run: pixi run -e ${{ matrix.env }} typecheck
       - run: pixi run -e ${{ matrix.env }} test
 
+  clean-gates:
+    name: clean gates (no cache)
+    if: github.event_name == 'schedule'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: prefix-dev/setup-pixi@v0.8.1
+        with:
+          cache: false
+      - run: pixi install --locked
+      - run: pixi run lint
+      - run: pixi run fmt-check
+      - run: pixi run typecheck
+      - run: pixi run test
+
   # Independent of gates: the bioconda solve is slow and orthogonal to the
   # python matrix. Needs BOTH envs installed — the harness runs on the default
   # env's python and invokes .pixi/envs/default/bin/gapit while parity env

gapit-0.6.2/.github/workflows/release.yml

@@ -0,0 +1,54 @@
+name: Release
+
+on:
+  workflow_dispatch:
+  push:
+    tags: ["v*"]
+
+permissions:
+  contents: read
+  # OIDC token for PyPI trusted publishing (pypa/gh-action-pypi-publish).
+  id-token: write
+
+jobs:
+  gates:
+    name: release gates
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: prefix-dev/setup-pixi@v0.8.1
+        with:
+          cache: false
+      - run: pixi install --locked
+      - run: pixi run lint
+      - run: pixi run fmt-check
+      - run: pixi run typecheck
+      - run: pixi run test
+
+  publish:
+    needs: gates
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Build sdist and wheel
+        run: |
+          python -m pip install --upgrade pip build twine
+          python -m build
+
+      - name: Check distributions
+        run: python -m twine check dist/*
+
+      - name: Install the built wheel and smoke-test it
+        run: |
+          python -m pip install dist/*.whl
+          gapit --version
+          gapit schema report > /dev/null
+          gapit db path --json > /dev/null
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

{gapit-0.5.6 → gapit-0.6.2}/AGENTS.md

@@ -23,8 +23,8 @@ are we?** A modern, typed, testable Python tool that:
 - **Environment manager**: [pixi](https://pixi.sh) (conda-forge channel). Never use pip/conda
   directly; add dependencies to `pixi.toml` (`pixi add <pkg>` for conda, `pixi add --pypi <pkg>`
   for PyPI).
-- **Python**: 3.14 in the pixi dev env (current stable); the package declares
-  `requires-python = ">=3.11"` and CI tests 3.11 / 3.13 / 3.14.
+- **Python**: 3.12 in the pixi dev env; the package declares
+  `requires-python = ">=3.12,<3.13"` and CI tests Python 3.12.
 - **External binaries**: BLAST+ (`blastn`, `blastx`, `makeblastdb`, `blastdbcmd`) and `minimap2`
   (FASTQ read screening, SPEC.md §10), all from conda-forge/bioconda. Invoked only via `subprocess`
   with an argument list — never `shell=True`. Input normalization (fa/fq/gbk/embl, gz/bz2) is
@@ -33,6 +33,8 @@ are we?** A modern, typed, testable Python tool that:
   `parity` pixi env (prepend `.pixi/envs/parity/bin` to PATH for the differential test).
 - **Core libraries**: `click` (CLI), `pydantic` v2 (data models / JSON schema), `rich` (terminal
   output). No biopython — FASTA I/O is a small streaming parser we own.
+- **Optional engine**: `gseek` (`gapit[gseek]`, Python 3.12) behind `GAPIT_ENGINE`;
+  `legacy` remains the default and needs no gseek import.
 - **Quality gates**: `ruff` (lint + format), `basedpyright` (strict mode), `pytest`.
 
 ### Commands (pixi tasks)
@@ -40,6 +42,7 @@ are we?** A modern, typed, testable Python tool that:
 ```bash
 pixi run lint        # ruff check
 pixi run fmt         # ruff format
+pixi run fmt-check   # ruff format --check
 pixi run typecheck   # basedpyright --strict
 pixi run test        # pytest (unit, offline)
 pixi run gapit       # the CLI itself
@@ -54,23 +57,27 @@ Every change must leave `lint`, `typecheck`, and `test` green.
 ```
 gapit/
 ├── AGENTS.md            # this file
-├── PLAN.md              # development roadmap (phase-gated)
+├── CONTRIBUTING.md      # contributor workflow and gates
+├── SECURITY.md          # trust boundaries + vulnerability reporting
+├── CITATION.cff         # citation metadata
+├── PLAN.md              # historical development roadmap
 ├── SPEC.md              # frozen contract for the baseline surface (distilled from abricate)
 ├── pixi.toml
 ├── recipe/
 │   └── meta.yaml        # conda recipe (submission deferred)
 ├── .github/workflows/
-│   └── ci.yml           # gates matrix 3.11/3.13/3.14 + parity job
+│   └── ci.yml           # gates matrix Python 3.12 + parity job
 ├── src/gapit/
 │   ├── __init__.py
-│   ├── cli.py           # typer entrypoint: screen / summary / db / setupdb / schema / mcp
-│   ├── config.py        # datadir resolution, defaults, env vars
+│   ├── cli.py           # click entrypoint: screen / summary / db / setupdb / schema / mcp
+│   ├── config.py        # datadir resolution, first-run bootstrap, defaults, env vars
 │   ├── dispatch.py      # shared CLI dispatch (error envelope → exit codes) + --datadir option + --output target
 │   ├── proctools.py     # external-tool plumbing: argv subprocess runner + stderr notes
 │   ├── fasta.py         # streaming FASTA reader + shared gz/bz2 text opener
 │   ├── seqconvert.py    # native input normalization: fa/fq/gbk/embl (±gz/bz2) → FASTA
 │   ├── db.py            # database discovery, header parsing, makeblastdb wrapper
 │   ├── dbcodec.py       # gapit/v1 tagged-header codec (percent-encoded ids)
+│   ├── db_txn.py        # per-db write lock + staging build + rename publish
 │   ├── records.py       # records.jsonl truth store + gapit.manifest/1 provenance
 │   ├── dbbuild.py       # deterministic native-db build pipeline with self-check
 │   ├── clusterbuild.py  # cluster-db build pipeline: GBK/GFF → locus FASTA + features.json + typing copy
@@ -100,19 +107,23 @@ gapit/
 │   ├── paf.py           # PAF row parsing + interval arithmetic (minimap2 output boundary)
 │   ├── report.py        # Report model: the canonical in-memory result
 │   ├── engines.py       # shared OutputFormat / AlignerEnum (leaf module for the three engines)
+│   ├── gseek_engine.py  # optional gseek bridge (legacy/gseek/shadow) for BLAST/minimap2
+│   ├── progress.py      # stderr-bound rich batch progress bar
 │   ├── screening.py      # blastn screen use-case + the shared run_screen dispatcher
 │   ├── screening_cluster.py # cluster screen use-case: kind guards, typing wiring, rendering
 │   ├── screening_reads.py # minimap2 use-cases: --r1/--r2 reads + --aligner minimap2 assemblies
+│   ├── screening_reads_positional.py # positional FASTQ wildcard: pairing + per-sample screens
 │   ├── typing_input.py   # `gapit typing` input: screen TSV/CSV → per-FILE folded GeneCalls
 │   ├── typing_ops.py     # `gapit typing` use-case: resolve db, run the engine, render
 │   ├── reads.py          # FASTQ mode: minimap2 PAF parsing, coverage breadth/depth, presence
+│   ├── readpairs.py      # positional FASTQ filename → sample/lane inference
 │   ├── summary.py       # summary core: parse report tables into a gene matrix
 │   ├── cmd_screen.py    # `gapit screen` CLI (registered from cli.py)
 │   ├── cmd_screen_positionals.py # screen positional guard: db-name-as-file usage error
 │   ├── cmd_screen_reads_args.py # reads-mode CLI arg rules (comma split + flag rejects)
 │   ├── cmd_summary.py   # `gapit summary` CLI (registered from cli.py)
 │   ├── cmd_typing.py    # `gapit typing` CLI (registered from cli.py)
-│   ├── cmd_db.py        # `gapit db` command group (fetch | list; registers the subcommands)
+│   ├── cmd_db.py        # `gapit db` command group (fetch/path/list/build/install/search/outdated)
 │   ├── cmd_db_install.py # `gapit db install`: SHA256-verified local-file install
 │   ├── cmd_db_build.py  # `gapit db build` CLI (custom FASTA → native db)
 │   ├── cmd_db_search.py # `gapit db search` CLI (records.jsonl lookup)
@@ -123,8 +134,10 @@ gapit/
 │   ├── errors.py        # typed errors + JSON error envelope
 │   ├── formats/
 │   │   ├── tsv.py       # abricate-compatible TSV/CSV (phenotype-free in every typing version)
+│   │   ├── csvutil.py   # CSV RFC 4180 + TSV cell sanitization (streaming renderers)
 │   │   ├── json.py      # versioned JSON (gapit.report/1 et al.)
 │   │   ├── reads_json.py # versioned JSON (gapit.reads/1, gapit.reads/2)
+│   │   ├── reads_tsv.py # streaming reads TSV/CSV table
 │   │   ├── md.py        # Markdown (human + agent readable, YAML frontmatter)
 │   │   ├── cluster.py   # gapit.cluster/1 models + JSON/TSV renderers (typed variant)
 │   │   ├── cluster2.py  # gapit.cluster/2 models + JSON/TSV renderers (multi-scheme phenotypes map)
@@ -179,15 +192,15 @@ This is what distinguishes gapit from abricate. Treat it as a public API.
   byte-identically), then `gapit typing result.tsv [-D|-f|-o|-q]` designates from the screen
   table (`gapit.typing_result/1`; cluster dbs refuse — their typing rides the screen).
   No typing MCP tool yet (future work).
-- **DB acquisition** `[gapit-extension]`: `gapit db fetch|list|search|outdated|build|install` —
+- **DB acquisition** `[gapit-extension]`: `gapit db fetch|path|list|search|outdated|build|install` —
   database fetch from the provider catalog (every provider downloads from upstream at fetch
   time — nothing catalog-side is bundled inside the wheel beyond the sanctioned exceptions,
   a license-compliance requirement: CARD/VFDB terms are non-commercial and Kaptive is GPL,
-  so none of those may ride an MIT distribution; the sanctioned exceptions are the six
+  so none of those may ride an MIT distribution; the sanctioned exceptions are the seven
   content-provenance-audited bundles under `src/gapit/data/dbs/` — the public-domain
-  `ecoli_dec` and `lm_doumith` panels, three permissively licensed provider snapshots
-  (`ncbi` public domain, `resfinder` Apache-2.0, `ecoh` BSD-3-Clause), and the MIT
-  `vp_ok` typed cluster snapshot — a pure built-in in the ecoli_dec/lm_doumith pattern: no
+  `ecoli_dec`, `lm_doumith`, and `vc_serogroup` panels, three permissively licensed
+  provider snapshots (`ncbi` public domain, `resfinder` Apache-2.0, `ecoh` BSD-3-Clause),
+  and the MIT `vp_ok` typed cluster snapshot — a pure built-in in the ecoli_dec/lm_doumith pattern: no
   provider module, no `db fetch`; vpautils OAgc+CPSgc merged content, MIT
   (GPL/NC content is never bundled) — materialized into the datadir on first use via
   `bundled.py` (gene bundles replay the gene pipeline, the cluster bundle the cluster
@@ -202,11 +215,16 @@ This is what distinguishes gapit from abricate. Treat it as a public API.
   MIT-licensed `vp_ok` typed cluster bundle (OAgc+CPSgc merged: the committed typing asset
   installs a 3-scheme `gapit.typing/2` doc so one screen renders `gapit.cluster/2` with
   the composed `O:K` serotype, zero network from first install), and SHA256-verified
-  local-file install.
+  local-file install. Datadir precedence is `--datadir`, `$GAPIT_DATADIR`,
+  `$XDG_DATA_HOME/gapit/db` (absolute XDG only), then `~/.local/share/gapit/db`;
+  `gapit db path [--json]` introspects it without creating, while `db list`/`setupdb`
+  bootstrap a missing root. See `docs/datadir.md` for the database anatomy, sharing and
+  Docker policy.
 - **MCP** `[gapit-extension]`: `gapit mcp` / `gapit-mcp` stdio server exposing nine tools:
   read-only `screen` (incl. `aligner minimap2` assembly survey), `screen_reads` (FASTQ via
-  minimap2), `summary`, `schema`, `db_list`, `db_search`, `db_outdated`, plus the datadir-mutating
-  `db_fetch` (installs databases from the provider catalog; may download) and `db_build` (writes a custom db);
+  minimap2), `summary`, `schema`, `db_search`, `db_outdated`, plus `db_list` (may create the
+  empty datadir on first use), and the datadir-mutating `db_fetch` (installs databases from
+  the provider catalog; may download) and `db_build` (writes a custom db);
   tool failures carry the `gapit.error/1` envelope.
 - **stdout purity**: data on stdout, diagnostics on stderr, always. `--quiet` only affects stderr.
 - **Self-description**: `gapit --version --json`, `gapit db list --json`, `gapit schema` — an agent
@@ -253,7 +271,7 @@ Growth tiers past the budget — split rather than inflate the ceiling:
   gene, %identity, %coverage). **From v0.5.0 this is a regression reference for the frozen
   baseline surface, not a release gate for new work** — new features need not check abricate
   and may diverge deliberately.
-- Full suite + CI matrix 3.11/3.13/3.14; parity byte-diff is opt-in via the `parity` pixi env.
+- Full suite + CI matrix Python 3.12; parity byte-diff is opt-in via the `parity` pixi env.
   Offline tests stay fast.
 
 ## 8. Domain knowledge (abridged; full detail in SPEC.md)

{gapit-0.5.6 → gapit-0.6.2}/CHANGELOG.md

@@ -5,8 +5,128 @@ All notable changes to gapit are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this
 project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.6.2] - 2026-10-26
+
+### Changed
+
+- **Version-only release.** No functional changes since 0.6.1; this release aligns
+  the package, pixi lock, docs examples, goldens, CITATION, and conda recipe on the
+  requested 0.6.2 version and republishes the 0.6.1 hardening set.
+
+## [0.6.1] - 2026-10-26
+
+### Fixed
+
+- **Fresh-install datadir bootstrap.** `gapit db list` (CLI, `--json`, and the MCP
+  `db_list` tool) no longer fails with `DATADIR_NOT_FOUND` when the datadir has never
+  been created: it bootstraps the resolved root and lists the catalog/bundled rows, so
+  a fresh `pip install gapit` can be inspected before any database is installed.
+  `gapit setupdb` likewise creates a missing datadir before materializing bundles.
+  Commands that consume installed databases keep the strict `DATADIR_NOT_FOUND`
+  semantics (bundled-name `screen`/`typing` still bootstrap on first use).
+- **Shared-datadir writes are serialized and published atomically.** Database builds now
+  write into a hidden staging directory and swap it into place only after success; a
+  per-database `flock` prevents concurrent `db fetch`/`db build` writers from interleaving.
+  Hidden staging/backup/lock entries are ignored by database discovery. New datadirs and
+  staging directories are private (`0700`). Readers see the old database for the whole
+  build, then the new one after the swap (a short rename window remains).
+- **CLI flag routing.** `--all-genes` is now forwarded to the minimap2 assembly
+  path, and reads-only flags (`--read-type`, `--min-breadth`, `--all-genes`) are
+  usage errors on blastn/cluster modes instead of being silently ignored.
+- **Fragment merging is record-aware.** The merge key now includes database,
+  accession, and subject length; zero aligned lengths no longer divide by zero.
+- **Cluster build rejects duplicate IDs.** Duplicate locus IDs and duplicate
+  gene IDs within a locus fail `BUILD_INVALID` before any artifact is written.
+- **Typing configuration is table-safe and compose-safe.** Compose schemes need
+  at least one placeholder; phenotype/fallback/notes reject tab/CR/LF; malformed
+  documents fail as `TYPING_MALFORMED` instead of crashing at screen time.
+- **Table output is delimiter-safe.** TSV cells neutralize embedded tab/CR/LF
+  (screen, reads, cluster, summary, typing, db list/search/outdated); product
+  tabs and cluster DB names are rejected at build time.
+- **Gene/cluster builds no longer require minimap2.** Its manifest version is
+  optional; `makeblastdb_version` now records the actual makeblastdb version.
+  BLAST indexes build under a temporary prefix and replace the old index only on
+  success.
+- **gseek parity fixes.** Missing gseek binaries map back to
+  `MISSING_DEPENDENCY`/exit 3, and the minimap2 index policy cache invalidates
+  when the target content changes.
+- **Reads suffix consistency.** `.fastq.gzip`/`.fq.gzip` now pair correctly.
+- **cluster/2 column collisions.** Schemes such as `o_group` and `o` fall back
+  to full evidence prefixes, keeping every column unique.
+- **Summary params completeness.** `gapit.summary/1` records `metrics`, so `-ic`
+  can be reproduced from the JSON document.
+- **Schema accuracy.** The never-serialized `PhenotypeDetail.winner` is omitted
+  from the public cluster schema.
+- **Read/metadata errors are typed.** OSError/UnicodeDecodeError in manifests,
+  records, floors, features, bundled metadata, fofn, summary/typing tables is
+  wrapped as the documented input/database error.
+- **Archive safety.** ZIP extraction rejects escaping members and caps expanded
+  size; plain-HTTP provider warnings survive `--quiet`.
+- **MCP parity.** Protocol version negotiation, CSV formats, and
+  `db_build` `kind`/`typing`/`floors` now match the CLI.
+- **Release safety.** The release workflow now runs the full gates, `twine check`,
+  and a wheel install smoke test before publishing.
+- **Documentation:** new `docs/datadir.md` (+zh), complete database catalog rows,
+  fixed anchors/links, historical `PLAN.md`, synced AGENTS tree, and new
+  `CONTRIBUTING.md`/`SECURITY.md`/`CITATION.cff`.
+- **Provider transport.** VFDB and Victors moved to HTTPS after verifying their
+  endpoints; BacMet2 remains HTTP-only and keeps the always-on warning.
+- **Shared setupdb locking.** `setupdb` index rebuilds now take the same per-database
+  writer lock as fetch/build; gseek's index-policy cache is lock-protected and drops
+  stale keys for a replaced target.
+- **GFF coordinate validation.** `end < 1` and `start > end` now raise typed
+  `GENE_OUT_OF_BOUNDS` instead of a pydantic `ValidationError`.
+- **`db list` index visibility.** `gapit.dblist/1` entries carry `indexed`, so an
+  installed-but-unindexed database is distinguishable from a ready one.
+- **MCP `all_genes`.** The screen/screen_reads tools expose the reads-table flag.
+- **Typing Markdown safety.** Source paths in YAML frontmatter are quoted.
+- **Fuzz/CI hardening.** Added deterministic parser fuzz smoke tests and a weekly
+  cache-free CI canary in addition to the release gates.
+
+### Added
+
+- **`gapit db path [--json]`.** Prints the effective datadir without creating it;
+  `--json` emits `gapit.datadir/1` (`datadir`, `exists`, `source` = `cli|env|xdg|default`).
+- **XDG Base Directory support.** Datadir precedence is now `--datadir`,
+  `$GAPIT_DATADIR`, `$XDG_DATA_HOME/gapit/db` (absolute XDG only), then
+  `~/.local/share/gapit/db`.
+
+### Changed
+
+- **gapit now uses `gseek.compat` for `GAPIT_ENGINE`.** The shared switch
+  replaces gapit's duplicate mode parser and shadow fallback; legacy remains
+  the default and gapit still owns its result mapping.
+
+- **Python 3.12 is now the supported version.** Drop the 3.11/3.13/3.14
+  environments from pixi/CI and align the package metadata, recipe, and docs
+  with gside/gmlst (`>=3.12,<3.13`).
+
+## [0.6.0] - 2026-10-25
+
+### Added
+
+- **Optional gseek engine for BLAST and minimap2.** `GAPIT_ENGINE=legacy|gseek|shadow`
+  routes contig/protein screening, reads, and cluster minimap2 through
+  [gseek](https://github.com/indexofire/gseek). `legacy` remains the default and is
+  retained on every path; `shadow` logs coarse differences and returns the legacy
+  result. gseek requires Python >= 3.12 and is available through `gapit[gseek]` or
+  the pixi `default` environment.
+- **FASTQ reads parity:** vfdb/card reads/1 and reads/2 are byte-identical to
+  legacy on a 3.07M-pair sample. gseek 0.8.1 allows duplicate minimap2 targets
+  (VFDB); the bridge uses `index="none"` for CARD's >255-character target names;
+  gseek 0.9.0 skips manifest-only query record counts.
+
 ## [0.5.6] - 2026-10-20
 
+### Added
+
+- **Bundled `vc_serogroup`: V. cholerae O1/O139 serogrouping + ctxA toxigenicity.** A
+  gene-kind bundle handed over from gside: `wbeN` (O1 wbe cluster) and `wbfR` (O139 wbf
+  cluster) as an exact-set `serogroup` scheme, plus the ctxA toxigenicity marker; O1 and
+  O139 exclude each other by construction, so a double call flags a mixed sample. Ships
+  from the wheel with zero network (the seventh bundled database), with floors and a
+  `gapit.typing/2` document.
+
 ### Changed
 
 - **`gapit.cluster/2` TSV/CSV reshaped: per-scheme winner evidence replaces the packed

gapit-0.6.2/CITATION.cff

@@ -0,0 +1,18 @@
+cff-version: 1.2.0
+message: "If you use gapit in research, please cite this software."
+title: "gapit: mass screening of contigs and reads for known genes"
+type: software
+authors:
+  - name: "indexofire"
+repository-code: "https://github.com/indexofire/gapit"
+url: "https://indexofire.github.io/gapit/"
+license: MIT
+version: "0.6.2"
+date-released: "2026-10-26"
+keywords:
+  - bioinformatics
+  - genomics
+  - antimicrobial resistance
+  - virulence factors
+  - BLAST
+  - minimap2