OperonDBS 0.7.2__tar.gz → 0.8.0__tar.gz

{operondbs-0.7.2 → operondbs-0.8.0}/.github/workflows/publish.yml

@@ -74,7 +74,8 @@ jobs:
           CIBW_BUILD: "cp310-* cp311-* cp312-* cp313-* cp314-*"
           CIBW_SKIP: "*-musllinux_*"
           CIBW_TEST_COMMAND: >-
-            python -c "import operon; import operon.qc_module._parsers;
+            python -c "import operon; import operon.qc._parsers;
+            import operon.qc._alignment;
             print(operon.__version__)" && operon --help
       - uses: actions/upload-artifact@v7
         with:

operondbs-0.8.0/.github/workflows/test.yml

@@ -0,0 +1,51 @@
+name: test
+
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+
+# A newer push supersedes an in-flight run of the same ref.
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  # Documentation is cheap to build and independent of the test matrix, so it
+  # runs as its own job in parallel instead of gating the ten test jobs.
+  docs:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+      - uses: actions/setup-python@v7
+        with:
+          python-version: "3.12"
+          cache: pip
+          cache-dependency-path: pyproject.toml
+      - run: python -m pip install --upgrade pip
+      - run: python -m pip install -e '.[docs]'
+      - run: sphinx-build -W --keep-going -b html docs docs/_build/html
+
+  pytest:
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ ubuntu-latest, macos-latest ]
+        python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v6
+      - uses: actions/setup-python@v7
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+          cache-dependency-path: pyproject.toml
+      - run: python -m pip install --upgrade pip
+      - run: python -m pip install -e '.[test]'
+      - run: python -m pytest -n 2 --dist loadfile
+      - if: ${{ !cancelled() }}
+        uses: codecov/codecov-action@v7
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}
+          report_type: test_results

{operondbs-0.7.2 → operondbs-0.8.0}/.gitignore

@@ -5,7 +5,8 @@ dist/
 __pycache__/
 *.py[cod]
 *.so
-operon/qc_module/_parsers.c
+operon/qc/_parsers.c
+operon/qc/_alignment.c
 *.egg-info/
 *.sqlite
 *.sqlite-wal
@@ -14,6 +15,7 @@ logs/
 raw/
 standardized/
 qc/
+!operon/qc/
 analysis/
 reports/
 releases/
@@ -26,3 +28,6 @@ junit.xml
 htmlcov/
 /site/
 /docs/_build/
+/workflows/
+/HPC/
+.matrix/

{operondbs-0.7.2 → operondbs-0.8.0}/AGENTS.md

@@ -26,8 +26,8 @@ principle-to-implementation mapping.
 
 Current version markers (must stay consistent across code and docs):
 
-- `operon` 0.7.0 (`pyproject.toml`)
-- database schema 2.9 (`operon/database.py`, `SCHEMA_VERSION`)
+- `operon` 0.8.0 (`pyproject.toml`)
+- database schema 2.11 (`operon/database.py`, `SCHEMA_VERSION`)
 - metadata schema 1.4 (`operon/schema.py`, `METADATA_SCHEMA_VERSION`)
 
 The project is licensed AGPL-3.0-or-later (`LICENSE` at the repo root).
@@ -36,17 +36,27 @@ The project is licensed AGPL-3.0-or-later (`LICENSE` at the repo root).
 
 - `operon/` — the Python package (CLI entry points: `operon/cli.py`,
   `operon/__main__.py`; console script `operon = operon.cli:main`).
-  - `operon/adapters/` — external source adapters (currently NCBI Datasets,
-    offline-first: JSON/JSONL, ZIP, or unpacked directories, plus optional
-    online download).
-  - `operon/qc_module/` — streaming FASTA/FASTQ/GFF3/protein parsers and
-    built-in QC stages. `parsers.py` is the pure-Python reference
+  - `operon/adapters/` — external source adapters: NCBI Datasets
+    (offline-first: JSON/JSONL, ZIP, or unpacked directories, plus optional
+    online download) and TimeTree (query-cache-only REST client caching the
+    verbatim responses of exact queries under `adapters_cache/timetree/`;
+    mirroring or redistribution is forbidden by TimeTree's terms).
+  - `operon/qc/` — home for all QC functionality: streaming
+    FASTA/FASTQ/GFF3/protein parsers and built-in QC stages, plus alignment
+    QC (`alignment.py`, the pure-Python multiple-alignment QC reference
+    implementation behind `operon alignment-qc`, and `_alignment.pyx`, its
+    Cython production backend; parity enforced by
+    `tests/regression/test_cython_alignment_parity.py`). `parsers.py` is the
+    pure-Python reference
     implementation; `_parsers.pyx` is the Cython-accelerated build of the
-    same API (compiled in place as `operon.qc_module._parsers`). The Cython
+    same API (compiled in place as `operon.qc._parsers`). The Cython
     module is the required production backend; the pure-Python module is the
     behavioral reference used by regression tests. Both must produce
     identical metrics and error messages (enforced by
-    `tests/regression/test_cython_parser_parity.py`).
+    `tests/regression/test_cython_parser_parity.py`). `measure.py` backs
+    `operon qc-measure`, a project-independent measurement-only path whose
+    JSON payload can be imported back through `operon import-qc` (the remote
+    built-in QC workflow).
   - `operon/execution.py` — execution backends for external commands:
     `local` subprocess, `slurm` (sbatch submit + squeue poll), and `ssh`
     (Paramiko; HPC head nodes and cloud VMs, optionally through remote
@@ -96,9 +106,18 @@ The project is licensed AGPL-3.0-or-later (`LICENSE` at the repo root).
     NCBI Taxonomy snapshots and coverage denominators), `release.py` +
     `export.py` (immutable releases and selective exports), `lifecycle.py`
     (audited reversible entity retirement), `lineage.py` (adopting external
-    workflow outputs), `backup.py`, `reports.py`, `table_import.py`,
+    workflow outputs), `sequence_tools.py` (alignment-driven domain
+    extraction and sequence selection), `classify.py` (versioned
+    sequence_classification profiles labelling `sequence_labels`),
+    `fanout.py` (data-derived fan-out of registered sequence files into
+    per-unit FASTAs under `analysis/derived/`, selected downstream by recipe
+    `file_role_prefix`), `timetree.py` (TimeTree query/calibration CLI
+    group backed by the query-cache adapter), `backup.py`, `reports.py`,
+    `table_import.py`,
     `import_wizard.py`, `entity_view.py`, `environment.py`
-    (execution-environment capture), `shutdown.py` (graceful SIGINT/SIGTERM
+    (execution-environment capture with at-capture redaction; recipe
+    `environment_policy` governs environment-aware cache reuse),
+    `shutdown.py` (graceful SIGINT/SIGTERM
     handling), `ncbi_reconcile.py` (development-era adapter anomaly repair),
     `demo.py` (deterministic synthetic demo project), `errors.py`,
     `utils.py`.
@@ -112,6 +131,9 @@ The project is licensed AGPL-3.0-or-later (`LICENSE` at the repo root).
   (`.readthedocs.yaml`).
 - `benchmarks/` — representative entity sets for QC performance diagnostics
   (see `docs/*/operations/qc-performance.md`).
+- `scripts/` — local developer tooling; `setup-test-matrix.sh` and
+  `run-test-matrix.sh` build and drive the uv-managed Python 3.10–3.14
+  matrix under .matrix/ (see `docs/*/contributor/development-testing.md`).
 
 ## Setup, test, and build
 
@@ -123,17 +145,22 @@ python3 -m venv .venv && source .venv/bin/activate
 python -m pip install -e '.[dev]'   # runtime + pytest + Cython + Sphinx;
                                     # also compiles the qc parsers extension
 
-python -m pytest                    # full suite (coverage gate: 90% branch)
+python -m pytest                    # full suite (coverage gate: >=95% combined
+                                    # line+branch, and >=95% branch coverage)
 python -m pytest tests/unit         # by category: unit / integration /
                                     # regression / compatibility
 
+python -m pytest --lf -q --no-cov   # iterate: last failures, no coverage
+python -m pytest -n 4 --dist loadfile   # parallel full suite (xdist, in the dev/test extras)
+scripts/run-test-matrix.sh          # whole suite on 3.10-3.14 concurrently
+
 python setup.py build_ext --inplace # rebuild only the Cython extension
 
 sphinx-build -W --keep-going -b html docs docs/_build/html  # strict docs build
 ```
 
 Run the relevant test category after any change; run the full suite before
-considering work done. CI (`.github/workflows/deploy.yml`) runs pytest on
+considering work done. CI (`.github/workflows/test.yml`) runs pytest on
 Python 3.10–3.14 and the strict Sphinx build. Releases are published
 exclusively to PyPI; see `docs/*/contributor/pypi-release.md`.
 
@@ -170,6 +197,11 @@ asks.
   exists only for pre-1.0 databases and is scheduled for removal at the 1.0
   release; check it before touching `operon/database.py` migrations or the
   NCBI adapter's schema-upgrade path.
+- The `operon.qc` module uses CFFI (specifically, Cython), so static analysis
+  tools, such as `flake8`, will report "undefined reference" errors (F821).
+  Using `# noqa: F821` behind the line **ONLY** when importing `operon.qc`
+  and an F821 error is likely to occur. Globally ignoring F821 is **STRICKLY
+  PROHIBITED**.
 
 ## Documentation sync
 
@@ -186,7 +218,7 @@ change:
 - Contributor-facing processes → `docs/*/contributor/`; navigation →
   `docs/*/index.md`
 
-Version markers in docs (`operon` 0.7.0, database schema 2.9, metadata
+Version markers in docs (`operon` 0.8.0, database schema 2.11, metadata
 schema 1.4) must match `pyproject.toml` and the code. Do not write the
 current values literally in Markdown sources: use the `myst_substitutions`
 references `{{ operon_version }}`, `{{ db_schema }}`, and

{operondbs-0.7.2 → operondbs-0.8.0}/MANIFEST.in

@@ -14,4 +14,5 @@ prune docs/_build
 recursive-include operon *.py *.pyx *.tcss
 recursive-include operon/tui/assets *.png *.z *.md
 recursive-include tests *.py
-exclude operon/qc_module/_parsers.c
+exclude operon/qc/_parsers.c
+exclude operon/qc/_alignment.c

{operondbs-0.7.2 → operondbs-0.8.0}/OperonDBS.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: OperonDBS
-Version: 0.7.2
+Version: 0.8.0
 Summary: Operon the Database System: Archive, Quality-Control, Organize, Analyze and Release Your Bio-Data
 Author-email: hyli360 <lihuanyu2003@gmail.com>
 License-Expression: AGPL-3.0-or-later
@@ -8,30 +8,26 @@ Project-URL: Repository, https://github.com/HYLi360/Operon
 Project-URL: Issues, https://github.com/HYLi360/Operon/issues
 Classifier: Operating System :: MacOS
 Classifier: Operating System :: POSIX :: Linux
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3 :: Only
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: aiohttp>=3.9
 Requires-Dist: biopython>=1.85
-Requires-Dist: PyYAML>=6.0
 Requires-Dist: paramiko>=3.4
+Requires-Dist: PyYAML>=6.0
 Requires-Dist: questionary>=2.1
 Requires-Dist: requests>=2.32
 Requires-Dist: textual>=8.0
 Provides-Extra: test
 Requires-Dist: coverage>=7.0; extra == "test"
+Requires-Dist: cython>=3.0; extra == "test"
 Requires-Dist: paramiko>=3.4; extra == "test"
-Requires-Dist: pytest-cov>=7.0; extra == "test"
 Requires-Dist: pytest>=8.0; extra == "test"
+Requires-Dist: pytest-cov>=7.0; extra == "test"
+Requires-Dist: pytest-xdist>=3.8; extra == "test"
 Requires-Dist: setuptools>=77; extra == "test"
 Requires-Dist: textual>=8.0; extra == "test"
-Requires-Dist: cython>=3.0; extra == "test"
 Provides-Extra: docs
 Requires-Dist: Sphinx<9,>=7.4; extra == "docs"
 Requires-Dist: myst-parser<5,>=4.0; extra == "docs"
@@ -40,8 +36,10 @@ Requires-Dist: tomli>=2.0; python_version < "3.11" and extra == "docs"
 Provides-Extra: dev
 Requires-Dist: coverage>=7.0; extra == "dev"
 Requires-Dist: cython>=3.0; extra == "dev"
-Requires-Dist: pytest-cov>=7.0; extra == "dev"
 Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-cov>=7.0; extra == "dev"
+Requires-Dist: pytest-xdist>=3.8; extra == "dev"
+Requires-Dist: setuptools>=77; extra == "dev"
 Requires-Dist: tomli>=2.0; python_version < "3.11" and extra == "dev"
 Requires-Dist: Sphinx<9,>=7.4; extra == "dev"
 Requires-Dist: myst-parser<5,>=4.0; extra == "dev"

{operondbs-0.7.2 → operondbs-0.8.0}/OperonDBS.egg-info/SOURCES.txt

@@ -50,8 +50,10 @@ docs/en/guides/file-archiving.md
 docs/en/guides/index.md
 docs/en/guides/metadata-import.md
 docs/en/guides/ncbi-datasets.md
+docs/en/guides/phylogenetics-workflow.md
 docs/en/guides/qc-profiles.md
 docs/en/guides/remote-execution.md
+docs/en/guides/remote-first.md
 docs/en/guides/remote-storage.md
 docs/en/guides/taxonomy-coverage.md
 docs/en/guides/troubleshooting.md
@@ -101,8 +103,10 @@ docs/zh/guides/file-archiving.md
 docs/zh/guides/index.md
 docs/zh/guides/metadata-import.md
 docs/zh/guides/ncbi-datasets.md
+docs/zh/guides/phylogenetics-workflow.md
 docs/zh/guides/qc-profiles.md
 docs/zh/guides/remote-execution.md
+docs/zh/guides/remote-first.md
 docs/zh/guides/remote-storage.md
 docs/zh/guides/taxonomy-coverage.md
 docs/zh/guides/troubleshooting.md
@@ -127,6 +131,7 @@ docs/zh/reference/recipe-parsers-examples.md
 operon/__init__.py
 operon/__main__.py
 operon/backup.py
+operon/classify.py
 operon/cli.py
 operon/config.py
 operon/coverage.py
@@ -134,9 +139,11 @@ operon/database.py
 operon/demo.py
 operon/entity_view.py
 operon/environment.py
+operon/environment_capture.py
 operon/errors.py
 operon/execution.py
 operon/export.py
+operon/fanout.py
 operon/files.py
 operon/import_wizard.py
 operon/lifecycle.py
@@ -149,17 +156,23 @@ operon/remotes.py
 operon/reports.py
 operon/rules.py
 operon/schema.py
+operon/sequence_tools.py
 operon/shutdown.py
 operon/table_import.py
 operon/taxonomy.py
+operon/timetree.py
 operon/tools.py
 operon/utils.py
 operon/workflow.py
 operon/adapters/__init__.py
 operon/adapters/ncbi_datasets.py
-operon/qc_module/__init__.py
-operon/qc_module/_parsers.pyx
-operon/qc_module/parsers.py
+operon/adapters/timetree.py
+operon/qc/__init__.py
+operon/qc/_alignment.pyx
+operon/qc/_parsers.pyx
+operon/qc/alignment.py
+operon/qc/measure.py
+operon/qc/parsers.py
 operon/tui/__init__.py
 operon/tui/actions.py
 operon/tui/app.py
@@ -189,50 +202,75 @@ tests/integration/__init__.py
 tests/integration/test_analysis_resume.py
 tests/integration/test_analysis_shutdown.py
 tests/integration/test_analysis_tools.py
+tests/integration/test_classify_sequences.py
 tests/integration/test_execution_backends.py
+tests/integration/test_fanout_analyze.py
 tests/integration/test_lineage_cascade.py
 tests/integration/test_ncbi_datasets_adapter.py
 tests/integration/test_pipeline_and_release.py
 tests/integration/test_python_packaging.py
+tests/integration/test_qc_remote_measure.py
 tests/integration/test_taxonomy_coverage.py
 tests/regression/__init__.py
 tests/regression/test_correctness.py
+tests/regression/test_cython_alignment_parity.py
 tests/regression/test_cython_parser_parity.py
 tests/regression/test_parser_semantics.py
 tests/unit/__init__.py
+tests/unit/test_alignment_qc.py
+tests/unit/test_classify.py
 tests/unit/test_cli_edges.py
 tests/unit/test_config_workflow_edges.py
+tests/unit/test_core_coverage_more.py
 tests/unit/test_coverage_edges.py
 tests/unit/test_database_edges_more.py
 tests/unit/test_docs_versions.py
 tests/unit/test_environment.py
+tests/unit/test_environment_capture.py
 tests/unit/test_execution.py
 tests/unit/test_execution_edges.py
 tests/unit/test_export.py
+tests/unit/test_fanout.py
 tests/unit/test_files_edges.py
+tests/unit/test_follow.py
 tests/unit/test_import_backup_show.py
 tests/unit/test_import_wizard_edges.py
 tests/unit/test_lifecycle.py
 tests/unit/test_lineage.py
+tests/unit/test_misc_coverage_more.py
+tests/unit/test_ncbi_coverage_gaps.py
 tests/unit/test_ncbi_edge_cases.py
 tests/unit/test_ncbi_reconcile_edges.py
 tests/unit/test_parser_edge_paths.py
 tests/unit/test_qc_and_rules.py
 tests/unit/test_qc_edges_more.py
+tests/unit/test_qc_measure.py
+tests/unit/test_qc_remote_only.py
 tests/unit/test_recipe_history.py
+tests/unit/test_remote_taxonomy_more.py
+tests/unit/test_remote_transport_edges.py
 tests/unit/test_remotes.py
 tests/unit/test_remotes_edges.py
 tests/unit/test_rules_schema_edges.py
+tests/unit/test_schema_2_10.py
+tests/unit/test_schema_2_11.py
 tests/unit/test_schema_2_9.py
 tests/unit/test_schema_and_metadata.py
+tests/unit/test_sequence_tools.py
+tests/unit/test_sequences.py
 tests/unit/test_shutdown.py
 tests/unit/test_support_edges.py
 tests/unit/test_table_import_edges.py
 tests/unit/test_taxonomy_edges.py
+tests/unit/test_timetree.py
+tests/unit/test_timetree_snapshots.py
 tests/unit/test_tools_edges.py
+tests/unit/test_tools_execution_more.py
 tests/unit/test_tui.py
 tests/unit/test_tui_config.py
+tests/unit/test_tui_flows_more.py
 tests/unit/test_tui_publish.py
+tests/unit/test_tui_screens_more.py
 tests/unit/test_tui_splash_terminal.py
 tests/unit/test_tui_writes.py
 tests/unit/test_views_release_reports_edges.py

{operondbs-0.7.2 → operondbs-0.8.0}/OperonDBS.egg-info/requires.txt

@@ -1,7 +1,7 @@
 aiohttp>=3.9
 biopython>=1.85
-PyYAML>=6.0
 paramiko>=3.4
+PyYAML>=6.0
 questionary>=2.1
 requests>=2.32
 textual>=8.0
@@ -9,8 +9,10 @@ textual>=8.0
 [dev]
 coverage>=7.0
 cython>=3.0
-pytest-cov>=7.0
 pytest>=8.0
+pytest-cov>=7.0
+pytest-xdist>=3.8
+setuptools>=77
 Sphinx<9,>=7.4
 myst-parser<5,>=4.0
 sphinx-rtd-theme<4,>=3.0
@@ -28,9 +30,10 @@ tomli>=2.0
 
 [test]
 coverage>=7.0
+cython>=3.0
 paramiko>=3.4
-pytest-cov>=7.0
 pytest>=8.0
+pytest-cov>=7.0
+pytest-xdist>=3.8
 setuptools>=77
 textual>=8.0
-cython>=3.0

{operondbs-0.7.2 → operondbs-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: OperonDBS
-Version: 0.7.2
+Version: 0.8.0
 Summary: Operon the Database System: Archive, Quality-Control, Organize, Analyze and Release Your Bio-Data
 Author-email: hyli360 <lihuanyu2003@gmail.com>
 License-Expression: AGPL-3.0-or-later
@@ -8,30 +8,26 @@ Project-URL: Repository, https://github.com/HYLi360/Operon
 Project-URL: Issues, https://github.com/HYLi360/Operon/issues
 Classifier: Operating System :: MacOS
 Classifier: Operating System :: POSIX :: Linux
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3 :: Only
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: aiohttp>=3.9
 Requires-Dist: biopython>=1.85
-Requires-Dist: PyYAML>=6.0
 Requires-Dist: paramiko>=3.4
+Requires-Dist: PyYAML>=6.0
 Requires-Dist: questionary>=2.1
 Requires-Dist: requests>=2.32
 Requires-Dist: textual>=8.0
 Provides-Extra: test
 Requires-Dist: coverage>=7.0; extra == "test"
+Requires-Dist: cython>=3.0; extra == "test"
 Requires-Dist: paramiko>=3.4; extra == "test"
-Requires-Dist: pytest-cov>=7.0; extra == "test"
 Requires-Dist: pytest>=8.0; extra == "test"
+Requires-Dist: pytest-cov>=7.0; extra == "test"
+Requires-Dist: pytest-xdist>=3.8; extra == "test"
 Requires-Dist: setuptools>=77; extra == "test"
 Requires-Dist: textual>=8.0; extra == "test"
-Requires-Dist: cython>=3.0; extra == "test"
 Provides-Extra: docs
 Requires-Dist: Sphinx<9,>=7.4; extra == "docs"
 Requires-Dist: myst-parser<5,>=4.0; extra == "docs"
@@ -40,8 +36,10 @@ Requires-Dist: tomli>=2.0; python_version < "3.11" and extra == "docs"
 Provides-Extra: dev
 Requires-Dist: coverage>=7.0; extra == "dev"
 Requires-Dist: cython>=3.0; extra == "dev"
-Requires-Dist: pytest-cov>=7.0; extra == "dev"
 Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-cov>=7.0; extra == "dev"
+Requires-Dist: pytest-xdist>=3.8; extra == "dev"
+Requires-Dist: setuptools>=77; extra == "dev"
 Requires-Dist: tomli>=2.0; python_version < "3.11" and extra == "dev"
 Requires-Dist: Sphinx<9,>=7.4; extra == "dev"
 Requires-Dist: myst-parser<5,>=4.0; extra == "dev"

{operondbs-0.7.2 → operondbs-0.8.0}/docs/conf.py

@@ -17,7 +17,7 @@ REPO_ROOT = DOCS_DIR.parent
 
 project = "Operon"
 author = "Project Operon Development Group"
-copyright = f"2026, {author}. All Rights Reserved."
+copyright = f"2026, {author}. All Rights Reserved"
 
 
 def _package_version() -> str:

{operondbs-0.7.2 → operondbs-0.8.0}/docs/en/architecture/extensibility.md

@@ -4,10 +4,10 @@
 
 The built-in source adapter currently covers NCBI Datasets; sources such as ENA remain part of the future extension boundary. Taxonomy coverage currently supports only NCBI Taxonomy; GTDB and the NCBI↔GTDB crosswalk are not yet implemented. Built-in QC covers file level, generic sequence basics for non-genome FASTA (CDS/protein), reads basics, assembly structure, and annotation structure. BUSCO is natively integrated through directory output and a JSON summary parser; tools without a parser yet — QUAST, Merqury, Kraken2, CheckM2, and similar — can still be integrated through `run-external` + `import-qc`. Downstream comparative-genomics analysis is done by external workflows in `analysis/`; `operon` is responsible for data admission, provenance, and publication.
 
-The contract between downstream workflows and the database is a closed loop formed by `operon export` and `operon adopt`: export materializes the selected entities by file identity into a `data/<entity_type>/<entity_id>/<filename>` layout, accompanied by `manifest.tsv` (with SHA-256 recomputed over the materialized bytes), a `qc.tsv` QC long-table snapshot, `checksums.sha256`, and `provenance.json` (the input-side manifest); after consuming this artifact set, the external workflow uses adopt to re-register derived artifacts as first-class manifest members — materialized under `analysis/adopted/<entity_id>/`, inheriting the ingest idempotency/conflict invariants, with file-to-file lineage edges recorded in `file_lineage` (the output-side manifest). Adopted products can be QC'd, evaluated, exported, released, and selected by `analyze` as inputs of downstream recipes (cascading analysis). Downstream workflows should read and write through this contract instead of reading the database directly. Orchestration of cascading workflows (dependency graphs, parallelism, retries) belongs to workflow managers such as snakemake/nextflow; `operon` is responsible for data admission, lineage, and publication, while `run-pipeline` only covers simple single-file chaining. The batch adopt manifest format is described in the [external analysis guide](../guides/external-analysis.md). Export is semantically complementary to release: release targets publication (QC-gated, immutable snapshot), while export targets analysis inputs (arbitrary selection criteria, materialized on demand).
+The contract between downstream workflows and the database is a closed loop formed by `operon export` and `operon adopt`: export materializes the selected entities by file identity into a `data/<entity_type>/<entity_id>/<filename>` layout, accompanied by `manifest.tsv` (with SHA-256 recomputed over the materialized bytes), a `qc.tsv` QC long-table snapshot, `checksums.sha256`, and `provenance.json` (the input-side manifest); after consuming this artifact set, the external workflow uses adopt to re-register derived artifacts as first-class manifest members — materialized under `analysis/adopted/<entity_id>/`, inheriting the ingest idempotency/conflict invariants, with file-to-file lineage edges recorded in `file_lineage` (the output-side manifest). Adopted products can be QC'd, evaluated, exported, released, and selected by `analyze` as inputs of downstream recipes (cascading analysis). Downstream workflows should read and write through this contract instead of reading the database directly. Orchestration of cascading workflows (dependency graphs, parallelism, retries) belongs to workflow managers such as snakemake/nextflow; `operon` is responsible for data admission, lineage, and publication, while `run-pipeline` only covers simple single-file chaining. One carve-out on the admission side: when the number of downstream analysis units is data-dependent (for example subfamilies whose count is known only after classification), admitting those units is `operon`'s responsibility — `operon fanout` materializes and registers the per-unit files with lineage under `analysis/derived/`; orchestrating the runs over them still belongs to the executor and the workflow engine. The batch adopt manifest format is described in the [external analysis guide](../guides/external-analysis.md). Export is semantically complementary to release: release targets publication (QC-gated, immutable snapshot), while export targets analysis inputs (arbitrary selection criteria, materialized on demand).
 
 Deduplication is implemented by layer: byte-level duplication is already guaranteed by SHA-256 idempotence (same entity and role with identical bytes returns the same `FIL_`; different bytes are explicitly rejected); sequence-level duplication (normalized sequence digests / refget-style summaries) and biological near-duplication (Mash/ANI/k-mer similarity, duplicate clusters, and representative selection) are extension directions — they can be performed with external tools in `analysis/` with results written back to `qc_results`, and the representative-selection rules themselves should also be versioned.
 
 For scale, SQLite WAL plus indexes suits metadata on the order of millions of rows, and all sequence parsing is streaming; if inode sharing is explicitly acceptable, hard links can be used for `standardized/` or releases.
 
-Execution backends extend along the abstraction in `execution.py`: `local`, `slurm`, and `ssh` are provided today, and a new backend only needs to implement the same executor interface to plug into `run-external`/`analyze`. Cloud-vendor SDKs (AWS Batch, GCP Batch, etc.) and Slurm array jobs are not yet supported; remote storage currently consists of SFTP mirrors only, and object storage (S3 and the like) is likewise an extension direction.
+Execution backends extend along the abstraction in `execution.py`: `local`, `slurm`, and `ssh` are provided today, and a new backend only needs to implement the same executor interface to plug into `run-external`/`analyze`. Cloud-vendor SDKs (AWS Batch, GCP Batch, etc.) and Slurm array jobs are not yet supported; remote storage currently consists of SFTP mirrors only, and object storage (S3 and the like) is likewise an extension direction. Data-dependent fan-out does not require array jobs: `operon fanout` admits each unit as its own manifest file, and `analyze` with a `file_role_prefix` recipe then submits one job per unit through any backend.

{operondbs-0.7.2 → operondbs-0.8.0}/docs/en/architecture/external-analysis.md

@@ -10,10 +10,12 @@ External programs such as BLAST/HMMER/BUSCO no longer require hand-assembled com
 4. Computes the reference database identity (single-file SHA-256 / directory fingerprint / explicit checksum);
 5. Looks up a completed cache by `analysis_name + file_id + input SHA + parameter fingerprint + tool version + database identity`, skipping on a hit; on a miss, enters the second-level continuation: if an old `completed` job for the same `(analysis, file_id)` has a matching input content hash and its recorded output artifact passes byte-level verification, that output is adopted (a new `completed` row is inserted under the current fingerprint, traced in the `changes` audit table, with status `adopted`); only otherwise is the job recomputed;
 6. On a miss, launches the program via `conda run`, a container prefix, or a direct path; both file and directory outputs must exist and be non-empty, with stdout/stderr written to disk;
-7. Computes the file or directory content hash, parses top hits or the BUSCO JSON summary into `analysis_hits`/`analysis_results`, and syncs the same-named metrics into `qc_results`.
+7. Computes the file or directory content hash, parses top hits or the BUSCO JSON summary into `analysis_hits`/`analysis_results`, and syncs the same-named metrics into `qc_results`. Parsers with coordinates (`blast_tabular`, `hmmer_domtblout`, `rpsbproc_tabular`) additionally write every parsed hit to `analysis_alignments` as a structured row (query/subject IDs, hit rank, query/subject intervals, e-value, bitscore, percent identity, unmapped columns in `extra_json`), untruncated by `max_hits_per_query`; `report analysis --hits` reads this table.
 
 Directories use a deterministic tree hash composed of relative paths, empty directories, file sizes/contents, and symlink targets. `database_mode: mutable_cache` serves shared caches such as BUSCO's that gradually download lineages, identified by an explicit `database_version` as the logical version; immutable reference databases keep the default `reference` content identity.
 
+A recipe may replace the single command with a `commands` chain (mutually exclusive with `arguments`). All rendered steps execute in order through the same executor and parent-tool `run_method` (one recipe, one environment — a deliberate capability limitation), under one run record and one `analysis_jobs` row. The first command is the recipe's logical owner: the job's `tool_version` and the version component of cache identity come from it — from the first block's own `version_args`/`version_pattern` when declared, otherwise from the tool-level probe (recipe loading forces the two to target the same program). A step invoking the parent `executable` inherits its already-probed version; another program may declare its own `version_args` and `version_pattern`, which are run through the same launcher/executor. Unconfigured secondary programs are explicitly recorded as version `unknown`; version flags are never guessed. Each step receives its own `logs/<run_id>.step<N>.stdout.log` / `.stderr.log`, and its `argv`, `executable`, `tool_version`, `tool_version_raw`, `version_source`, `version_command`, and `exit_code` are recorded in `execution_details.steps`. The first non-zero step aborts the chain and fails the job with a `step N/M failed` error. Intermediate artifacts live in a deterministic `${work_dir}` scratch directory (`<output_name>.work` next to the output), which is deleted and rebuilt before execution and removed after success or failure (`--keep-partial` keeps it). Expected-output validation, content hashing, and result parsing run once, after the last step. On SSH backends with a remote root the work directory is mapped to the remote root and created there like any other project path. The canonical example couples `rpsblast -outfmt 11` (ASN.1 archive) with independently versioned `rpsbproc`, whose tabular report is parsed by `rpsbproc_tabular`: domain rows from the `DATA`/`SESSION`/`QUERY`/`DOMAINS` blocks become full `analysis_alignments` rows (the query is the QUERY definition line, the subject is the accession, coordinates come from the `from`/`to` columns, and hit type/PSSM IDs/short name stay in `extra_json`), with EAV hits truncated to `max_hits_per_query` as usual. Every probed step version is mixed into the cache fingerprint: upgrading any step's program invalidates the exact cache even when the recipe text and the primary tool version are unchanged, with verified-output adoption applying as usual.
+
 Actual execution of external commands is taken over by the backend abstraction in `execution.py`; `run_external_command` selects a backend through `get_executor(project, backend)`:
 
 - `local` (default): the original local subprocess behavior, completely unchanged;
@@ -22,13 +24,17 @@ Actual execution of external commands is taken over by the backend abstraction i
 
 All three backends share the same provenance and correctness contract: exit codes, start/end times, and logs are written to `workflow_runs` and `logs/workflow.jsonl` as usual; SQLite additionally stores the executor, scheduler job ID, and resource/script details; success determination and input/output SHA-256 verification are unchanged; tool version probing on non-`local` backends also goes through the same backend. An individual recipe can override same-named fields of `execution.slurm` with a `slurm:` mapping (e.g. adjusting memory/time for BUSCO alone).
 
-Execution environment capture (schema 2.8, `environment.py`): all three backends probe the execution environment on every run. The normalized JSON document is stored in the `execution_environments` table; its primary key `environment_id` is the SHA-256 of that normalized JSON (content-addressed, so identical documents deduplicate automatically), and both `workflow_runs` and `analysis_jobs` reference it through their own `environment_id` column.
+Execution environment capture (`environment.py`, `environment_capture.py`) stores normalized JSON in `execution_environments`, addressed by `environment_id`. Runs and analysis jobs reference that document; each command-chain step also records its own environment ID. No database migration is needed for the extended JSON document (`capture_schema: 1`). The document is redacted at capture time, before storage: the hostname is replaced by `sha256:` plus the first 16 hex characters of its SHA-256, and the `$HOME` prefix of path-like values (`PATH`, `CONDA_PREFIX`, `VIRTUAL_ENV`, the conda prefix) is replaced by `~`. Remote probes transfer the raw values (probe files are written with umask 077) and the controlling Python process redacts them before storage; a transient `home` probe key feeds the replacement and is then discarded, never entering the stored document or any fingerprint. The `environment_id` and sub-fingerprints are computed from the redacted document, so identical configurations captured on different hosts deduplicate to one record. Documents recorded before redaction was introduced are content-addressed and immutable and are not migrated; new captures produce new environment IDs.
+
+The local, Slurm and SSH backends probe immediately before the payload. Slurm probes after `setup_commands` on the compute node; direct SSH probes in the payload shell's working directory. Recognized `conda`, `mamba` and `micromamba run` launchers run the probe through the same launcher and target name/prefix. Reading `conda-meta/*.json` requires no Python or package-manager CLI inside the target environment. Direct programs inherit the executor environment. Known opaque shell/container launchers and unsupported manager options are labelled `unsupported_launcher`, with `capture_scope: executor_only`; their outer environment is not claimed as the tool's Conda environment. Arbitrary custom wrappers are not introspected.
+
+Conda snapshots retain package name, version, build, subdir, dependencies, archive URL and available SHA-256/MD5. Complete inventories include an `@EXPLICIT` reconstruction specification (SHA-256 preferred, MD5 fallback). Package fingerprints exclude installation prefixes and hostnames. Credential-bearing URL components (userinfo, `/t/` tokens and queries) are removed. Private channels may consequently require credentials supplied separately during restoration. Pip-installed distribution names detected through `INSTALLER` metadata are reported separately; neither pip packages, editable installs, manually modified files nor custom activation scripts are restored by the Conda specification. The package inventory describes original archives, not a verification of installed bytes. Empty, malformed, interrupted or unresolvable inventories cannot be exported as complete specifications.
+
+Separate `system_fingerprint` and `hardware_fingerprint` cover OS/kernel/architecture, distribution and glibc, and CPU model/features, total memory and available NVIDIA GPU model/driver/compute capability respectively. Hostname, PATH, prefix, CPU affinity and runtime settings do not enter those two fingerprints. Selected locale, timezone and threading/device variables and CPU affinity are retained as context. These are Linux-oriented probes: missing CPU/GPU data is explicitly marked unavailable (GPU absence and probe failure are not distinguished); no dynamic metrics, GPU UUIDs or serial numbers are collected. Hardware fingerprints describe observed fields and do not certify hardware equivalence when fields are unavailable.
 
-- `local`: collects hostname, OS/kernel/architecture, Python and `operon` versions, the environment variables `PATH`, `CONDA_PREFIX`, `CONDA_DEFAULT_ENV`, `VIRTUAL_ENV`, `SINGULARITY_NAME`, `APPTAINER_NAME`, `container`, and a docker probe, all directly;
-- `slurm`: embeds the probe in the sbatch script, writes the result to `<run_id>.env` inside the job, and reads it back — the captured environment is the compute node's, not the submission host's;
-- `ssh`: direct mode runs the probe on the remote host through paramiko; remote Slurm mode embeds and retrieves the same job-side probe so it also records the compute node rather than the SSH login node.
+Probes are best-effort, bounded by 30 seconds (NVIDIA query: 5 seconds), and require a POSIX shell, `base64` and common system utilities. GNU `timeout` bounds the probe when the host provides it; elsewhere — macOS ships no `timeout` — a POSIX watcher enforces the same 30-second bound, so a host without it still captures instead of reporting `unavailable`. Linux-only data sources such as `/proc` simply leave their fields unavailable. `capture_status` distinguishes complete, partial, failed, unavailable and unsupported launcher captures. Earlier documents remain readable and may lack these fields. Controller-only legacy probes remain available as fallback; their Python version must not be interpreted as the target interpreter version.
 
-A failed probe leaves the run's `environment_id` NULL without raising an error or affecting the run; rows from before 2.8 are likewise NULL.
+Environment fingerprints also drive a recipe-level reuse policy: the optional recipe field `environment_policy` takes `ignore`, `warn` (the default) or `strict`, and any other value is rejected at configuration validation. The field enters the parameter fingerprint only when it is explicitly set — the same mechanism as `hmmer_mode` and the alignment column-mapping keys — so setting or changing it invalidates the completed cache exactly once. On a completed-cache hit with a policy other than `ignore`, `operon` compares an environment-relevance fingerprint: the composite of the document's `system_fingerprint`, `hardware_fingerprint` and `conda.package_fingerprint`. Hostname, paths, CPU affinity and runtime threading variables are deliberately excluded — the thread count already participates in the parameter fingerprint, and affinity is transient scheduler state. If the fingerprints match, the cache is reused as usual; on a mismatch, `warn` reuses the cached result while recording a warning in the run details and printing it, and `strict` treats the hit as a miss and recomputes. Backends without a pre-job probe (`slurm` and remote Slurm) cannot compare before reuse, so `strict` degrades to `warn` and the downgrade is recorded in the run details. When either side's environment document lacks sub-fingerprints (captures recorded before database schema {{ db_schema }}), the comparison is recorded as `unavailable`: `warn` reuses as usual and `strict` likewise degrades to `warn`, so legacy provenance never causes spurious recomputation.
 
 Recipe versioning and snapshots (schema 2.9): a recipe gains an optional `version:` field (a positive integer, default 1; invalid values are rejected at configuration validation). As `analyze` processes each candidate file it records the current recipe together with the verbatim spec of its referenced tool into the `recipe_snapshots` table: the snapshot document is `{"recipe": <the recipe's raw mapping>, "tool": <the referenced tool spec's raw mapping>}`, content-addressed by the SHA-256 of its canonicalized JSON and deduplicated by `UNIQUE(recipe_name, recipe_version, recipe_sha256)` — so edits to the tool definition also produce a new snapshot, and cache hits record a snapshot of the current configuration as well. `analysis_jobs.recipe_snapshot_id` points back to the exact configuration that produced the job; jobs adopted during resume inherit the original job's snapshot id (they were produced by that configuration, not today's). Inspect them with `operon recipes list / history / show`; the QC-profile counterparts recorded in `qc_profiles` are inspected with `operon profiles history / show`. Restoration via the CLI is print-only in both cases: a human copies the output back into the configuration YAML, and the CLI never rewrites files in place. The audited alternative is the TUI Config screen, whose structured editors save every change as the next version with a new snapshot and can restore any recorded snapshot into the editor (saving it creates the next version); TUI recipe saves normalize `tools.yaml` formatting and drop hand-written comments. <!-- version-pin -->
 

{operondbs-0.7.2 → operondbs-0.8.0}/docs/en/architecture/files-and-storage.md

@@ -18,12 +18,12 @@ Guarantees of `ingest`:
 The `remotes:` section of `project.yaml` can configure one or more SFTP remote mirrors (`operon/remotes.py`), synchronizing manifest files to a remote without breaking the invariants of this section:
 
 - Plain files and directory artifacts are all verified by `sha256 + size_bytes`; when the server lacks `sha256sum`, SHA-256 is computed by streaming over SFTP, never degrading to size-only comparison; directories use exactly the same deterministic tree hash as locally (including empty directories and symlink targets);
-- The remote maintains an `operon-manifest.json` v2 manifest (project_id + relative_path → file_id/sha256/size/kind/synced_at). Manifest updates require the server to support the OpenSSH POSIX rename extension and are published via "unique temporary file + atomic replacement"; one push batch publishes the manifest exactly once, and read-modify-write is serialized through the remote atomic directory `.operon-manifest.lock`, so concurrent pushes from multiple control ends cannot lose entries;
+- The remote maintains an `operon-manifest.json` v2 manifest (project_id + relative_path → file_id/sha256/size/kind/synced_at) whose updates are published atomically and serialized between concurrent control ends;
 - All relative paths are root-constrained on both sides; absolute paths, `..`, and path escapes are rejected. The remote manifest's `project_id` and every entry identity must match the local SQLite;
 - Every transfer reuses `workflow_runs` for provenance (step `push:<name>` / `pull:<name>`); successful locations are also cached in `file_locations`;
-- push/pull/evict use per-item result semantics: a failed item records `error` and the rest of the batch continues; the CLI returns non-zero if any error exists;
-- After `pull` restores a locally missing file, `files.status` returns to `CHECKSUM_VERIFIED`, and this change is written to `changes` just like the status changes from `verify`/`evict`;
-- `ingest --source` also directly accepts `sftp://[user@]host[:port]/path` and `remote://<name>/<path>`; the latter must exist in the remote manifest and is identity-checked first, while the former is downloaded and assigned a fresh identity by ingest, then follows exactly the same archiving flow as a local file.
+- After `pull` restores a locally missing file, `files.status` returns to `CHECKSUM_VERIFIED`, and this change is written to `changes` just like the status changes from `verify`/`evict`.
+
+Manifest publication and locking, batch semantics and per-item exit codes, and `sftp://`/`remote://` ingest are command-level details; see [Remote storage commands](../reference/cli-remote.md) and [File and QC Commands](../reference/cli-files-qc.md#ingest).
 
 Paramiko is a core runtime dependency but remains lazily imported, so local-only commands do not initialize the SSH stack.