OperonDBS 0.8.0__tar.gz → 0.8.2__tar.gz

{operondbs-0.8.0 → operondbs-0.8.2}/.github/workflows/publish.yml

@@ -71,7 +71,7 @@ jobs:
       - run: python -m cibuildwheel --output-dir wheelhouse
         env:
           CIBW_ARCHS: ${{ matrix.architecture }}
-          CIBW_BUILD: "cp310-* cp311-* cp312-* cp313-* cp314-*"
+          CIBW_BUILD: "cp310-* cp311-* cp312-* cp313-* cp314-* cp315-*"
           CIBW_SKIP: "*-musllinux_*"
           CIBW_TEST_COMMAND: >-
             python -c "import operon; import operon.qc._parsers;

{operondbs-0.8.0 → operondbs-0.8.2}/.github/workflows/test.yml

@@ -32,20 +32,21 @@ jobs:
       fail-fast: false
       matrix:
         os: [ ubuntu-latest, macos-latest ]
-        python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
+        python-version: ["3.10", "3.11", "3.12", "3.13", "3.14", "3.15"]
     runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v6
       - uses: actions/setup-python@v7
         with:
+          allow-prereleases: true
+          check-latest: true
           python-version: ${{ matrix.python-version }}
           cache: pip
           cache-dependency-path: pyproject.toml
-      - run: python -m pip install --upgrade pip
-      - run: python -m pip install -e '.[test]'
-      - run: python -m pytest -n 2 --dist loadfile
-      - if: ${{ !cancelled() }}
-        uses: codecov/codecov-action@v7
+      - run: |
+          python -m pip install --upgrade pip
+          python -m pip install -e '.[test]'
+          python -m pytest
+      - uses: codacy/codacy-coverage-reporter-action@v1.3.0
         with:
-          token: ${{ secrets.CODECOV_TOKEN }}
-          report_type: test_results
+          project-token: ${{ secrets.CODACY_PROJECT_TOKEN }}

{operondbs-0.8.0 → operondbs-0.8.2}/.gitignore

@@ -31,3 +31,4 @@ htmlcov/
 /workflows/
 /HPC/
 .matrix/
+cobertura.xml

{operondbs-0.8.0 → operondbs-0.8.2}/AGENTS.md

@@ -26,7 +26,7 @@ principle-to-implementation mapping.
 
 Current version markers (must stay consistent across code and docs):
 
-- `operon` 0.8.0 (`pyproject.toml`)
+- `operon` 0.8.2 (`pyproject.toml`)
 - database schema 2.11 (`operon/database.py`, `SCHEMA_VERSION`)
 - metadata schema 1.4 (`operon/schema.py`, `METADATA_SCHEMA_VERSION`)
 
@@ -133,7 +133,9 @@ The project is licensed AGPL-3.0-or-later (`LICENSE` at the repo root).
   (see `docs/*/operations/qc-performance.md`).
 - `scripts/` — local developer tooling; `setup-test-matrix.sh` and
   `run-test-matrix.sh` build and drive the uv-managed Python 3.10–3.14
-  matrix under .matrix/ (see `docs/*/contributor/development-testing.md`).
+  matrix under .matrix/ (see `docs/*/contributor/development-testing.md`);
+  `defects.sh` appends to and queries the defect registry (`defects.yml`,
+  see "Defect reports" below).
 
 ## Setup, test, and build
 
@@ -152,7 +154,7 @@ python -m pytest tests/unit         # by category: unit / integration /
 
 python -m pytest --lf -q --no-cov   # iterate: last failures, no coverage
 python -m pytest -n 4 --dist loadfile   # parallel full suite (xdist, in the dev/test extras)
-scripts/run-test-matrix.sh          # whole suite on 3.10-3.14 concurrently
+scripts/run-test-matrix.sh          # whole suite on 3.10-3.15 concurrently
 
 python setup.py build_ext --inplace # rebuild only the Cython extension
 
@@ -160,12 +162,33 @@ sphinx-build -W --keep-going -b html docs docs/_build/html  # strict docs build
 ```
 
 Run the relevant test category after any change; run the full suite before
-considering work done. CI (`.github/workflows/test.yml`) runs pytest on
-Python 3.10–3.14 and the strict Sphinx build. Releases are published
-exclusively to PyPI; see `docs/*/contributor/pypi-release.md`.
+considering work done.
 
-Do not commit or perform other git mutations unless the user explicitly
-asks.
+CI (`.github/workflows/test.yml`) runs pytest on Python 3.10–3.15 and the strict
+Sphinx build. Releases are published exclusively to PyPI; see `docs/*/contributor/pypi-release.md`.
+
+The project uses `pytest-xdist` to parallel testing. Avoid sharing state between tests
+to prevent unexpected or random test results.
+
+> Running `python -m pytest` directly takes approx 35 seconds
+> (auto: 24 workers). Running the local test matrix takes 3 minutes
+> (6 jobs × 4 workers), or 4~5min (6 jobs × 2 workers in default).
+> 
+> Measured on Intel Core i7-13700HX (16c24t).
+
+## Git
+
+If the user requests or grants permission, do `git commit` after modify.
+Avoid including overly large changes in a single commit. For commit messages,
+follow the pattern used in the last five commits; otherwise, the Conventional
+Commits guidelines.
+
+**DO NOT** push any commit or tag unless the user **explicitly** requests that.
+
+If times out, assume the user is not present and that a GPG signature
+is required. Add `-c commit.gpgsign=false` behind `git` may help (However,
+re-signing will reset the commit hash, and corrupt the defects.yml. Please
+proceed with caution).
 
 ## Conventions
 
@@ -203,6 +226,22 @@ asks.
   and an F821 error is likely to occur. Globally ignoring F821 is **STRICKLY
   PROHIBITED**.
 
+## Defect reports
+
+Confirmed defects are tracked in the machine-readable registry
+`defects.yml` at the repository root (plus any `defects/*.yml` shards;
+`scripts/defects.sh` appends and queries records). The full process is in
+`docs/*/contributor/defect-tracking.md`; the binding rules are:
+
+1. **Register first.** When an audit or investigation confirms a defect,
+   append its `ODR-XXXX` record to `defects.yml` before the fix lands.
+2. **One commit per defect.** The fix and its regression tests ride in the
+   same commit; fill in `fix_commit` when committing.
+3. **Close the test loop.** Regression tests for a defect carry
+   `@pytest.mark.bug("ODR-XXXX")`, and every `fixed`/`verified` record
+   lists at least one such test; `tests/unit/test_defect_registry.py`
+   validates the registry schema and both directions of this closure.
+
 ## Documentation sync
 
 When you change behavior, CLI surface, configuration fields, or storage
@@ -218,7 +257,7 @@ change:
 - Contributor-facing processes → `docs/*/contributor/`; navigation →
   `docs/*/index.md`
 
-Version markers in docs (`operon` 0.8.0, database schema 2.11, metadata
+Version markers in docs (`operon` 0.8.2, database schema 2.11, metadata
 schema 1.4) must match `pyproject.toml` and the code. Do not write the
 current values literally in Markdown sources: use the `myst_substitutions`
 references `{{ operon_version }}`, `{{ db_schema }}`, and
@@ -235,6 +274,9 @@ Sphinx-rendered); update the list above when bumping.
 
 ## Special Note For Codex/ChatGPT
 
-Due to specific limitations of the sandbox environment, executing certain TUI test code may cause the system to freeze; this often occurs on Codex/ChatGPT. The specific reason is that when this test code runs in the sandbox, it may results a Textual/asyncio cleanup block, and reports "FAIL" due to timeout.
+Due to specific limitations of the sandbox environment, executing certain TUI
+test code may cause the system to freeze; this often occurs on Codex/ChatGPT.
+The specific reason is that when this test code runs in the sandbox, it may
+result a Textual/asyncio cleanup block, and reports "FAIL" due to timeout.
 
 If you are Codex/ChatGPT, please execute TUI-related test code OUTSIDE the sandbox.

{operondbs-0.8.0 → operondbs-0.8.2}/OperonDBS.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: OperonDBS
-Version: 0.8.0
+Version: 0.8.2
 Summary: Operon the Database System: Archive, Quality-Control, Organize, Analyze and Release Your Bio-Data
 Author-email: hyli360 <lihuanyu2003@gmail.com>
 License-Expression: AGPL-3.0-or-later
@@ -46,9 +46,14 @@ Requires-Dist: myst-parser<5,>=4.0; extra == "dev"
 Requires-Dist: sphinx-rtd-theme<4,>=3.0; extra == "dev"
 Dynamic: license-file
 
-# Operon
+# Operon the Database System
 
-[![test status](https://github.com/HYLi360/Operon/actions/workflows/test.yml/badge.svg)](https://github.com/HYLi360/Operon/actions/workflows/test.yml) [![codecov](https://codecov.io/gh/HYLi360/Operon/branch/master/graph/badge.svg?token=BC4LD8UPL2)](https://codecov.io/gh/HYLi360/Operon)
+[![test status](https://github.com/HYLi360/Operon/actions/workflows/test.yml/badge.svg)](https://github.com/HYLi360/Operon/actions/workflows/test.yml)
+[![RtD badge](https://img.shields.io/badge/Read-the_Document-blue?style=flat)](https://operonproject.readthedocs.io/)
+![PyPI Version](https://img.shields.io/pypi/v/OperonDBS?link=https%3A%2F%2Fpypi.org%2Fproject%2FOperonDBS%2F)
+![GitHub commit activity](https://img.shields.io/github/commit-activity/t/HYLi360/Operon?logo=github)
+[![Codacy Code quality Badge](https://app.codacy.com/project/badge/Grade/47d652651ff645ff86fa0646768a08b3)](https://app.codacy.com/gh/HYLi360/Operon/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade)
+[![Codacy Coverage Badge](https://app.codacy.com/project/badge/Coverage/47d652651ff645ff86fa0646768a08b3)](https://app.codacy.com/gh/HYLi360/Operon/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_coverage)
 
 A Python-based, **file-based database** designed for large-scale genomic data, used for archiving, quality control, analysis, and deterministic automation.
 

{operondbs-0.8.0 → operondbs-0.8.2}/OperonDBS.egg-info/SOURCES.txt

@@ -33,6 +33,7 @@ docs/en/architecture/overview.md
 docs/en/architecture/qc-and-rules.md
 docs/en/architecture/release-lifecycle.md
 docs/en/architecture/taxonomy-coverage.md
+docs/en/contributor/defect-tracking.md
 docs/en/contributor/development-testing.md
 docs/en/contributor/documentation-deployment.md
 docs/en/contributor/index.md
@@ -75,6 +76,7 @@ docs/en/reference/index.md
 docs/en/reference/recipe-fields.md
 docs/en/reference/recipe-overview.md
 docs/en/reference/recipe-parsers-examples.md
+docs/en/reference/resolved-issues.md
 docs/zh/index.md
 docs/zh/overview.md
 docs/zh/architecture/extensibility.md
@@ -86,6 +88,7 @@ docs/zh/architecture/overview.md
 docs/zh/architecture/qc-and-rules.md
 docs/zh/architecture/release-lifecycle.md
 docs/zh/architecture/taxonomy-coverage.md
+docs/zh/contributor/defect-tracking.md
 docs/zh/contributor/development-testing.md
 docs/zh/contributor/documentation-deployment.md
 docs/zh/contributor/index.md
@@ -128,6 +131,7 @@ docs/zh/reference/index.md
 docs/zh/reference/recipe-fields.md
 docs/zh/reference/recipe-overview.md
 docs/zh/reference/recipe-parsers-examples.md
+docs/zh/reference/resolved-issues.md
 operon/__init__.py
 operon/__main__.py
 operon/backup.py
@@ -158,6 +162,7 @@ operon/rules.py
 operon/schema.py
 operon/sequence_tools.py
 operon/shutdown.py
+operon/sql.py
 operon/table_import.py
 operon/taxonomy.py
 operon/timetree.py
@@ -199,6 +204,7 @@ tests/helpers.py
 tests/compatibility/__init__.py
 tests/compatibility/test_python_support.py
 tests/integration/__init__.py
+tests/integration/test_analysis_array.py
 tests/integration/test_analysis_resume.py
 tests/integration/test_analysis_shutdown.py
 tests/integration/test_analysis_tools.py
@@ -216,6 +222,7 @@ tests/regression/test_correctness.py
 tests/regression/test_cython_alignment_parity.py
 tests/regression/test_cython_parser_parity.py
 tests/regression/test_parser_semantics.py
+tests/regression/test_sql_identifier_security.py
 tests/unit/__init__.py
 tests/unit/test_alignment_qc.py
 tests/unit/test_classify.py
@@ -224,6 +231,8 @@ tests/unit/test_config_workflow_edges.py
 tests/unit/test_core_coverage_more.py
 tests/unit/test_coverage_edges.py
 tests/unit/test_database_edges_more.py
+tests/unit/test_defect_registry.py
+tests/unit/test_defects_cli.py
 tests/unit/test_docs_versions.py
 tests/unit/test_environment.py
 tests/unit/test_environment_capture.py

{operondbs-0.8.0 → operondbs-0.8.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: OperonDBS
-Version: 0.8.0
+Version: 0.8.2
 Summary: Operon the Database System: Archive, Quality-Control, Organize, Analyze and Release Your Bio-Data
 Author-email: hyli360 <lihuanyu2003@gmail.com>
 License-Expression: AGPL-3.0-or-later
@@ -46,9 +46,14 @@ Requires-Dist: myst-parser<5,>=4.0; extra == "dev"
 Requires-Dist: sphinx-rtd-theme<4,>=3.0; extra == "dev"
 Dynamic: license-file
 
-# Operon
+# Operon the Database System
 
-[![test status](https://github.com/HYLi360/Operon/actions/workflows/test.yml/badge.svg)](https://github.com/HYLi360/Operon/actions/workflows/test.yml) [![codecov](https://codecov.io/gh/HYLi360/Operon/branch/master/graph/badge.svg?token=BC4LD8UPL2)](https://codecov.io/gh/HYLi360/Operon)
+[![test status](https://github.com/HYLi360/Operon/actions/workflows/test.yml/badge.svg)](https://github.com/HYLi360/Operon/actions/workflows/test.yml)
+[![RtD badge](https://img.shields.io/badge/Read-the_Document-blue?style=flat)](https://operonproject.readthedocs.io/)
+![PyPI Version](https://img.shields.io/pypi/v/OperonDBS?link=https%3A%2F%2Fpypi.org%2Fproject%2FOperonDBS%2F)
+![GitHub commit activity](https://img.shields.io/github/commit-activity/t/HYLi360/Operon?logo=github)
+[![Codacy Code quality Badge](https://app.codacy.com/project/badge/Grade/47d652651ff645ff86fa0646768a08b3)](https://app.codacy.com/gh/HYLi360/Operon/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade)
+[![Codacy Coverage Badge](https://app.codacy.com/project/badge/Coverage/47d652651ff645ff86fa0646768a08b3)](https://app.codacy.com/gh/HYLi360/Operon/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_coverage)
 
 A Python-based, **file-based database** designed for large-scale genomic data, used for archiving, quality control, analysis, and deterministic automation.
 

{operondbs-0.8.0 → operondbs-0.8.2}/README.md

@@ -1,6 +1,11 @@
-# Operon
-
-[![test status](https://github.com/HYLi360/Operon/actions/workflows/test.yml/badge.svg)](https://github.com/HYLi360/Operon/actions/workflows/test.yml) [![codecov](https://codecov.io/gh/HYLi360/Operon/branch/master/graph/badge.svg?token=BC4LD8UPL2)](https://codecov.io/gh/HYLi360/Operon)
+# Operon the Database System
+
+[![test status](https://github.com/HYLi360/Operon/actions/workflows/test.yml/badge.svg)](https://github.com/HYLi360/Operon/actions/workflows/test.yml)
+[![RtD badge](https://img.shields.io/badge/Read-the_Document-blue?style=flat)](https://operonproject.readthedocs.io/)
+![PyPI Version](https://img.shields.io/pypi/v/OperonDBS?link=https%3A%2F%2Fpypi.org%2Fproject%2FOperonDBS%2F)
+![GitHub commit activity](https://img.shields.io/github/commit-activity/t/HYLi360/Operon?logo=github)
+[![Codacy Code quality Badge](https://app.codacy.com/project/badge/Grade/47d652651ff645ff86fa0646768a08b3)](https://app.codacy.com/gh/HYLi360/Operon/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade)
+[![Codacy Coverage Badge](https://app.codacy.com/project/badge/Coverage/47d652651ff645ff86fa0646768a08b3)](https://app.codacy.com/gh/HYLi360/Operon/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_coverage)
 
 A Python-based, **file-based database** designed for large-scale genomic data, used for archiving, quality control, analysis, and deterministic automation.
 

{operondbs-0.8.0 → operondbs-0.8.2}/README_ZH.md

@@ -1,6 +1,11 @@
-# Operon
-
-[![test status](https://github.com/HYLi360/Operon/actions/workflows/test.yml/badge.svg)](https://github.com/HYLi360/Operon/actions/workflows/test.yml) [![codecov](https://codecov.io/gh/HYLi360/Operon/branch/master/graph/badge.svg?token=BC4LD8UPL2)](https://codecov.io/gh/HYLi360/Operon)
+# Operon the Database System
+
+[![test status](https://github.com/HYLi360/Operon/actions/workflows/test.yml/badge.svg)](https://github.com/HYLi360/Operon/actions/workflows/test.yml)
+[![RtD badge](https://img.shields.io/badge/Read-the_Document-blue?style=flat)](https://operonproject.readthedocs.io/)
+![PyPI Version](https://img.shields.io/pypi/v/OperonDBS?link=https%3A%2F%2Fpypi.org%2Fproject%2FOperonDBS%2F)
+![GitHub commit activity](https://img.shields.io/github/commit-activity/t/HYLi360/Operon?logo=github)
+[![Codacy Code quality Badge](https://app.codacy.com/project/badge/Grade/47d652651ff645ff86fa0646768a08b3)](https://app.codacy.com/gh/HYLi360/Operon/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade)
+[![Codacy Coverage Badge](https://app.codacy.com/project/badge/Coverage/47d652651ff645ff86fa0646768a08b3)](https://app.codacy.com/gh/HYLi360/Operon/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_coverage)
 
 一个基于 Python 的、面向大规模基因组数据的**基于文件的数据库**，用于归档、质控、分析与确定性自动化处理。
 

{operondbs-0.8.0 → operondbs-0.8.2}/docs/en/architecture/external-analysis.md

@@ -20,13 +20,13 @@ Actual execution of external commands is taken over by the backend abstraction i
 
 - `local` (default): the original local subprocess behavior, completely unchanged;
 - `slurm`: a local Slurm cluster. Generates a `<run_id>.sbatch` batch script under `logs/` (`--cpus-per-task` takes the thread count; optional `--time`/`--partition`/`--mem`, `extra_sbatch`, and `setup_commands`), submits with `sbatch --parsable`, polls `squeue` at `poll_interval`, and after the job disappears reads the `<run_id>.exitcode` file written by the script (falling back to `sacct` on failure); requires the project directory to be on a filesystem shared with the compute nodes;
-- `ssh`: executes on an SSH remote host (HPC head node / cloud VM) through the lazily imported Paramiko runtime dependency; with `execution.ssh.scheduler: slurm`, sbatch/squeue runs on the remote instead. Supports `remote_root` path mapping (empty means a shared filesystem); input files are uploaded over SFTP (identical content skipped, strict SHA-256/directory-tree hash, different content refused). When `storage_remote` is configured, REMOTE_ONLY inputs are consumed in place on the remote. Before a run, the exactly computed old remote output is removed; expected outputs are pulled back through temporary files and compared against the remote content again; an existing local output is accepted only when its content is identical. `storage_remote` and an explicit `remote_root` must point to the same root. One analysis batch uses a single lazy SSH client for version probing, remote input verification, database preflight, and all commands, closing it uniformly at the end. Direct commands run in a separate process group via util-linux `setsid --wait` (guaranteeing reliable exit-code return); on timeout, TERM/KILL is sent to the remote process group based on a restricted PID file, and when a termination request cannot be sent, the error and provenance state explicitly that the process may still be running. Remote Slurm polls strictly at `poll_interval` and retries the exitcode file that is briefly invisible after a job disappears.
+- `ssh`: executes on an SSH remote host (HPC head node / cloud VM) through the lazily imported Paramiko runtime dependency; with `execution.ssh.scheduler: slurm`, sbatch/squeue runs on the remote instead. Supports `remote_root` path mapping (empty means a shared filesystem); input files are uploaded over SFTP (identical content skipped, strict SHA-256/directory-tree hash, different content refused). When `storage_remote` is configured, REMOTE_ONLY inputs are consumed in place on the remote. Before a run, an existing remote output is renamed to a `<path>.operon-prev-<uuid>` backup rather than deleted; the backup is dropped once the run succeeds and the new output verifies, and a failed or interrupted run best-effort restores it in place. Expected outputs are pulled back through temporary files and compared against the remote content again; an existing local output is accepted only when its content is identical. `storage_remote` and an explicit `remote_root` must point to the same root. One analysis batch uses a single lazy SSH client for version probing, remote input verification, database preflight, and all commands, closing it uniformly at the end. Direct commands run in a separate process group via util-linux `setsid --wait` (guaranteeing reliable exit-code return); on timeout, TERM/KILL is sent to the remote process group based on a restricted PID file, and when a termination request cannot be sent, the error and provenance state explicitly that the process may still be running. Remote Slurm polls strictly at `poll_interval` and retries the exitcode file that is briefly invisible after a job disappears.
 
 All three backends share the same provenance and correctness contract: exit codes, start/end times, and logs are written to `workflow_runs` and `logs/workflow.jsonl` as usual; SQLite additionally stores the executor, scheduler job ID, and resource/script details; success determination and input/output SHA-256 verification are unchanged; tool version probing on non-`local` backends also goes through the same backend. An individual recipe can override same-named fields of `execution.slurm` with a `slurm:` mapping (e.g. adjusting memory/time for BUSCO alone).
 
 Execution environment capture (`environment.py`, `environment_capture.py`) stores normalized JSON in `execution_environments`, addressed by `environment_id`. Runs and analysis jobs reference that document; each command-chain step also records its own environment ID. No database migration is needed for the extended JSON document (`capture_schema: 1`). The document is redacted at capture time, before storage: the hostname is replaced by `sha256:` plus the first 16 hex characters of its SHA-256, and the `$HOME` prefix of path-like values (`PATH`, `CONDA_PREFIX`, `VIRTUAL_ENV`, the conda prefix) is replaced by `~`. Remote probes transfer the raw values (probe files are written with umask 077) and the controlling Python process redacts them before storage; a transient `home` probe key feeds the replacement and is then discarded, never entering the stored document or any fingerprint. The `environment_id` and sub-fingerprints are computed from the redacted document, so identical configurations captured on different hosts deduplicate to one record. Documents recorded before redaction was introduced are content-addressed and immutable and are not migrated; new captures produce new environment IDs.
 
-The local, Slurm and SSH backends probe immediately before the payload. Slurm probes after `setup_commands` on the compute node; direct SSH probes in the payload shell's working directory. Recognized `conda`, `mamba` and `micromamba run` launchers run the probe through the same launcher and target name/prefix. Reading `conda-meta/*.json` requires no Python or package-manager CLI inside the target environment. Direct programs inherit the executor environment. Known opaque shell/container launchers and unsupported manager options are labelled `unsupported_launcher`, with `capture_scope: executor_only`; their outer environment is not claimed as the tool's Conda environment. Arbitrary custom wrappers are not introspected.
+The local, Slurm and SSH backends probe immediately before the payload. The local probe is memoized in-process per (launcher, working directory) for 300 seconds (`_LOCAL_CAPTURE_TTL_SECONDS`), so a batch of commands through one launcher pays for one probe; failed captures are not cached. Slurm probes after `setup_commands` on the compute node; direct SSH probes in the payload shell's working directory. Recognized `conda`, `mamba` and `micromamba run` launchers run the probe through the same launcher and target name/prefix. Reading `conda-meta/*.json` requires no Python or package-manager CLI inside the target environment. Direct programs inherit the executor environment. Known opaque shell/container launchers and unsupported manager options are labelled `unsupported_launcher`, with `capture_scope: executor_only`; their outer environment is not claimed as the tool's Conda environment. Arbitrary custom wrappers are not introspected.
 
 Conda snapshots retain package name, version, build, subdir, dependencies, archive URL and available SHA-256/MD5. Complete inventories include an `@EXPLICIT` reconstruction specification (SHA-256 preferred, MD5 fallback). Package fingerprints exclude installation prefixes and hostnames. Credential-bearing URL components (userinfo, `/t/` tokens and queries) are removed. Private channels may consequently require credentials supplied separately during restoration. Pip-installed distribution names detected through `INSTALLER` metadata are reported separately; neither pip packages, editable installs, manually modified files nor custom activation scripts are restored by the Conda specification. The package inventory describes original archives, not a verification of installed bytes. Empty, malformed, interrupted or unresolvable inventories cannot be exported as complete specifications.
 
@@ -48,6 +48,16 @@ Whenever a backend cannot collect a metric the column stays NULL — no error is
 
 `run-external` provenance is aligned with `analyze`: `--tool NAME` matching a tool configured in `config/tools.yaml` triggers automatic version detection recorded as `tool_version` and `tool_version_raw` (a failed probe degrades to a warning and does not block the run); `--input PATH` (repeatable) declares input files whose per-file SHA-256 values are combined into `input_sha256`, with the full list recorded in `execution_details`; `--threads` records the thread count requested from the execution backend.
 
-Graceful shutdown (`shutdown.py`): while an `analyze` batch is running, SIGINT/SIGTERM handlers are installed; a signal is converted into `ShutdownRequested` (a `KeyboardInterrupt` subclass) raised on the main thread, and cleanup follows the normal exception path — `local` backend subprocesses start with `start_new_session` in their own process group, and on interruption/timeout the whole process group (including grandchildren) receives TERM then KILL; the `slurm` and remote-Slurm backends `scancel` the current job on interruption; the `ssh` direct backend reuses the timeout path's remote process-group TERM/KILL. The current `analysis_jobs` row is set to `interrupted` (it does not participate in completed-cache hits), half-finished output is deleted by default (`--keep-partial` keeps it), and the batch then terminates with exit code 130; a second signal during cleanup forces an immediate exit. A `RUNNING` row left behind by a SIGKILLed process is swept to `interrupted` at the next `analyze` startup, so resume semantics always hold.
+Graceful shutdown (`shutdown.py`): while an `analyze` batch is running, SIGINT/SIGTERM handlers are installed; a signal is converted into `ShutdownRequested` (a `KeyboardInterrupt` subclass) raised on the main thread, and cleanup follows the normal exception path — `local` backend subprocesses start with `start_new_session` in their own process group, and on interruption/timeout the whole process group (including grandchildren) receives TERM then KILL; the `slurm` and remote-Slurm backends `scancel` the current job on interruption; the `ssh` direct backend reuses the timeout path's remote process-group TERM/KILL. The current `analysis_jobs` row is set to `interrupted` (it does not participate in completed-cache hits), half-finished output is deleted by default (`--keep-partial` keeps it), and the batch then terminates with exit code 130; a second signal during cleanup forces an immediate exit. A `RUNNING` row of the same analysis left behind by a SIGKILLed process is swept to `interrupted` at the next `analyze` startup, so resume semantics always hold.
+
+## Slurm array submission
+
+On HPC clusters, submitting N files of one recipe as N separate scheduler jobs pays N times the queue and accounting overhead for what is logically one batch. When a recipe opts in (`slurm.array: true`), the selected executor implements array submission, and at least two files of one `analyze` batch miss the cache, `analyze` instead submits them as a single Slurm job array — one `sbatch`, one `squeue` poll loop — with each file occupying one array task. The `local` backend and direct SSH (`scheduler: none`) do not implement array submission, so any unmet condition falls back to the per-file submission path unchanged; for the SSH executor the capability exists only with `scheduler: slurm` (its `run_array` attribute is `None` otherwise).
+
+The mechanism mirrors the per-file path. `run_array` writes a tab-separated manifest (`logs/<batch_id>.array-manifest.tsv`) with one line per task — index, run_id, stdout/stderr/exitcode/probe paths, and the rendered command as the last field — and renders one array script (`render_slurm_array_script`) that dispatches line N to array task N via `sed -n "${SLURM_ARRAY_TASK_ID}p"`. The SBATCH header, `setup_commands`, the `cd` guard, the per-task environment probe, and the self-written exit-code file are shared with the single-job script; `slurm.array_concurrency: N` becomes the `%N` throttle on `--array=1-M%N`. Each task keeps its own `logs/<run_id>.*` files, so per-file provenance granularity is unchanged: every file still gets its own `workflow_runs` row, with `scheduler_job_id` recorded as `<array_id>_<task_index>` and the array id kept in the run details as `array_job_id`. Accounting is queried per task (`sacct -j <array_id>_<index>`), so the sacct signal-folding semantics (an OOM kill's `0:9` becomes 137) apply per task exactly as for single jobs. Through the SSH executor with a remote Slurm scheduler the same flow runs remotely: manifest, probe scripts, and array script are staged over SFTP, `sbatch`/`squeue` run on the remote host, and per-task logs, exit codes, and sacct accounting are pulled back.
+
+Array participation is a submission strategy, not an environment or parameter difference, so it never enters the parameter fingerprint or cache identity: a recipe shares the same completed cache with array submission on or off, and cache hits work identically either way. Because candidate order is deterministic and the manifest carries each task's rendered command, the task-to-file mapping is auditable and immune to resume/`--limit` drift.
+
+Interrupt bookkeeping contract: on KeyboardInterrupt the whole array is cancelled once with `scancel` and the exception propagates; the caller distinguishes finished tasks by which per-task `<run_id>.exitcode` files exist (the remote path pulls back any exit-code files that already exist before re-raising, so the same check works against local paths). Tasks without an exit-code file are marked `interrupted` — they never pollute the completed cache — and their partial outputs are removed under the usual per-file semantics (`--keep-partial` keeps them), so re-running the same command resumes exactly the unfinished files.
 
 For day-to-day use, see the [how-to guides](../guides/index.md); for the full contract of fields, placeholders, artifacts, database identity, caching, and parsers, see the [recipe configuration reference](../reference/recipe-overview.md).

operondbs-0.8.2/docs/en/contributor/defect-tracking.md

@@ -0,0 +1,71 @@
+# Defect tracking
+
+The current defect tracking of Operon records the confirmed defects in
+a machine-readable registry: `defects.yml` at the repository root,
+instead of external issue tracker. The registry is the single source
+of truth for defect reports and their disposition; [Resolved issues](../reference/resolved-issues.md)
+is the user-facing summary and keeps the historical K-series numbers
+alongside the ODR series.
+
+> K-series numbers are archived only in [Resolved Issues](../reference/resolved-issues.md).
+> The registry will not be updated with, or supplemented by K-series numbers.
+
+## Registry
+
+Every confirmed defect gets one record under `defects:` with a sequential
+`ODR-XXXX` id. Records load from `defects.yml` plus any `defects/*.yml`
+shards (sorted by name), so an overgrown registry can be split without
+changing tooling.
+
+Record fields:
+
+| Field | Content |
+|---|---|
+| `id` | `ODR-XXXX`, sequential, unique (validated) |
+| `title` | one-line summary |
+| `reported` | ISO date the defect was confirmed |
+| `introduced_in` | version or commit that introduced the defect, or `null` if undetermined |
+| `affected` | affected versions and the preconditions for triggering |
+| `severity` | `low` / `medium` / `high` / `critical` |
+| `component` | primary module or subsystem |
+| `status` | `open` → `confirmed` → `fixed` → `verified`; `wontfix` / `duplicate` are terminal |
+| `reproduction` | observed behavior and how it was reproduced |
+| `disposition` | what was done (or why no action) |
+| `fix_commit` | 40-hex commit id once the fix lands; `null` while pending (format-validated only) |
+| `fixed_in` | release version once shipped; `null` until then |
+| `regression_tests` | `path::test` list; mandatory for `fixed`/`verified` |
+
+Use `scripts/defects.sh` to append and query records:
+
+```bash
+scripts/defects.sh list [--status open] [--component tools]
+scripts/defects.sh show ODR-0001
+scripts/defects.sh add --title "..." --severity medium --component tools \
+    --reproduction "..."
+```
+
+The shell script is a small launcher for `scripts/defects.py`; it selects the
+project virtual environment when available. `list` uses a table on wide
+terminals and a compact per-record layout on narrow terminals. `show` formats
+long record fields as wrapped sections. Colors are enabled only for an
+interactive terminal and can be disabled with `NO_COLOR`.
+
+`add` allocates the next id and appends a `status: open` record. It keeps the
+existing registry schema and YAML storage format unchanged.
+
+## Rules
+
+1. **Register first.** When an audit or investigation confirms a defect,
+   append its record to `defects.yml` before the fix lands.
+2. **One commit per defect.** The fix and its regression tests ride in the
+   same commit; fill in `fix_commit` when committing.
+3. **Close the test loop.** Every regression test for a defect carries
+   `@pytest.mark.bug("ODR-XXXX")` (registered in `pyproject.toml`), and
+   every `fixed`/`verified` record lists at least one such test.
+   `tests/unit/test_defect_registry.py` validates the registry schema and
+   both directions of the marker closure, so a missing record, a dangling
+   marker, or a `fixed` record without a regression test fails the suite.
+   `python -m pytest -m bug` runs exactly the defect regressions.
+
+`verified` additionally requires `fix_commit` and `fixed_in`, and is set
+when the fix ships in a release.

{operondbs-0.8.0 → operondbs-0.8.2}/docs/en/contributor/development-testing.md

@@ -2,7 +2,8 @@
 
 ## Setup and test runs
 
-Install the repository checkout with the `dev` extra as described in [Installation](../getting-started/installation.md#install-from-the-repository); it adds pytest, Cython, and Sphinx. Then:
+Install the repository checkout with the `dev` extra as described in [Installation](../getting-started/installation.md#install-from-the-repository); it adds pytest, Cython, and
+Sphinx. Then:
 
 ```bash
 python -m pytest
@@ -16,36 +17,44 @@ python -m pytest tests/regression tests/compatibility
 sphinx-build -W --keep-going -b html docs docs/_build/html
 ```
 
-The pytest suite is organized into four categories — `unit`, `integration`, `regression`, `compatibility` — covering: Python 3.10 syntax and runtime gates, schema validation and controlled vocabularies, metadata round-trips and transaction rollback, stable IDs, default copy isolation, query read-only constraints, file-aware QC identity, profile/decision history, gzip FASTA recognition, assembly/annotation QC, rule decisions, idempotent ingest and conflict protection, checksum-tamper detection, the demo end-to-end pipeline and release verification, the NCBI Datasets adapter, wrapped BLAST/HMMER/BUSCO execution, directory artifacts, JSON summaries, conda run prefix parsing, cache hits/forced re-runs, result write-back, and input-tamper rejection.
-The taxonomy coverage integration tests additionally cover taxonomy source-package identity conflicts, profile type/content conflicts, exclusion rules, secondary TaxIDs, denominator/report idempotence, and that active metadata modifications do not affect the release-frozen scope.
+The pytest suite is organized into four categories — `unit`, `integration`, `regression`, `compatibility` — covering:
+Python 3.10 syntax and runtime gates, schema validation and controlled vocabularies, metadata round-trips and
+transaction rollback, stable IDs, default copy isolation, query read-only constraints, file-aware QC identity,
+profile/decision history, gzip FASTA recognition, assembly/annotation QC, rule decisions, idempotent ingest and conflict
+protection, checksum-tamper detection, the demo end-to-end pipeline and release verification, the NCBI Datasets adapter,
+wrapped BLAST/HMMER/BUSCO execution, directory artifacts, JSON summaries, conda run prefix parsing, cache hits/forced
+re-runs, result write-back, and input-tamper rejection.
+The taxonomy coverage integration tests additionally cover taxonomy source-package identity conflicts, profile
+type/content conflicts, exclusion rules, secondary TaxIDs, denominator/report idempotence, and that active metadata
+modifications do not affect the release-frozen scope.
+
+The project uses `pytest-xdist` to parallel testing. Avoid sharing state between tests to prevent unexpected or random
+test results.
 
 ## Fast local verification
 
-The full suite takes about seven minutes serially; the loop below aims to run it at most
-once per change.
+The full suite takes about seven minutes serially; the loop below aims to run it at most once per change.
 
 1. **Re-run only what failed.** `python -m pytest --lf -q --no-cov` replays pytest's
    last-failed cache before anything else.
 2. **Stop at the first failure** while iterating: `python -m pytest tests/unit -x -q --no-cov`,
    with the file or test you actually touched rather than a whole category.
-3. **Parallelize.** `python -m pytest -n 4 --dist loadfile` (pytest-xdist) runs the suite in
-   about a quarter of the time — roughly 7 minutes to under 2 on a 24-core workstation — and
-   `--dist loadfile` keeps each test file on one worker, which the Textual UI tests need.
-   It ships with the `test` and `dev` extras (`python -m pip install -e '.[dev]'`). Coverage is the expensive part:
-   iterate with `--no-cov` and measure once, at the end.
-4. **Cross-version matrix in one pass.** `scripts/setup-test-matrix.sh` creates uv-managed
-   CPython 3.10-3.13 interpreters plus one venv per version inside `.matrix/` (3.14 comes
-   from the project `.venv`), and `scripts/run-test-matrix.sh` runs all five concurrently:
+3. **Parallel execution.** On a 24-core workstation, running `python -m pytest -n 24 --dist loadfile` 
+   (with `pytest-xdist`) can reduce the execution time to 30–40 seconds. `--dist loadfile` ensures that each test file
+   is assigned to only one worker, which is required for Textual UI testing.
+   Coverages are the most resource-intensive: use `--no-cov` during iterations and run the test only once at the end.
+4. **Cross-version matrix in one pass.** `scripts/setup-test-matrix.sh` creates uv-managed CPython 3.10-3.15
+   interpreters plus one venv per version inside `.matrix/`. `scripts/run-test-matrix.sh` runs all six concurrently:
 
    ```bash
    scripts/setup-test-matrix.sh                      # once per machine
-   scripts/run-test-matrix.sh                        # whole suite, five versions, ~3 minutes
+   scripts/run-test-matrix.sh                        # whole suite, six versions, 2~3 minutes
    scripts/run-test-matrix.sh tests/unit -x          # extra pytest arguments are forwarded
-   MATRIX_JOBS=8 scripts/run-test-matrix.sh          # workers per version
+   MATRIX_JOBS=4 scripts/run-test-matrix.sh          # workers per version
    ```
 
    Each version logs to `.matrix/logs/<version>.log`, so a failure there can be re-run
-   directly with that interpreter. The scripts work on Linux and macOS; `.matrix/` is
+   directly with that interpreter. The scripts work on Linux and macOS. `.matrix/` is
    git-ignored.
 5. **Platform-specific code cannot be verified by one matrix job alone.** Linux and macOS
    differ in the system utilities the environment probe calls and in `pathlib`/`resource`
@@ -53,14 +62,12 @@ once per change.
    should carry a test that reproduces the other platform's condition on Linux — for
    example an empty `PATH` without GNU `timeout` — rather than depending on the macOS job
    to notice.
-6. **CI runs the matrix in parallel.** The strict documentation build is its own small
-   job, and the ten-job matrix (two workers per job, so a four-core runner is not
-   oversubscribed) starts immediately; use the local ladder above when you want a faster
-   verdict than the matrix can give.
 
 ## Special Note For Codex/ChatGPT
 
-Due to specific limitations of the sandbox environment, executing certain TUI test code may cause the system to freeze; this often occurs on Codex/ChatGPT. The specific reason is that when this test code runs in the sandbox, it may results a Textual/asyncio cleanup block, and reports "FAIL" due to timeout.
+Due to specific limitations of the sandbox environment, executing certain TUI test code may cause the system to freeze;
+this often occurs on Codex/ChatGPT. The specific reason is that when this test code runs in the sandbox, it may result 
+a Textual/asyncio cleanup block, and reports "FAIL" due to timeout.
 
 If you are Codex/ChatGPT, please execute TUI-related test code OUTSIDE the sandbox.
 
@@ -68,7 +75,10 @@ This information has also been updated in AGENTS.md.
 
 ## Coverage gate
 
-`python -m pytest` always measures coverage (`--cov=operon --cov-branch`, configured in `pyproject.toml`) and fails when the reported total drops below the `fail_under` threshold. The reported total is the combined figure `(covered lines + covered branches) / (valid lines + valid branches)`, so a change that adds only lines without their branches lowers it. Branch coverage must additionally stay at or above 90 % of all valid branches.
+`python -m pytest` always measures coverage (`--cov=operon --cov-branch`, configured in `pyproject.toml`) and fails when
+the reported total drops below the `fail_under` threshold. The reported total is the combined figure
+`(covered lines + covered branches) / (valid lines + valid branches)`, so a change that adds only lines without their
+branches lowers it. Branch coverage must additionally stay at or above 95% of all valid branches.
 
 Read the gaps with:
 

{operondbs-0.8.0 → operondbs-0.8.2}/docs/en/contributor/index.md

@@ -4,6 +4,7 @@
 - [Development and Testing](development-testing.md)
 - [PyPI Release](pypi-release.md)
 - [Documentation builds and Read the Docs deployment](documentation-deployment.md)
+- [Defect tracking](defect-tracking.md)
 
 ```{toctree}
 :hidden:
@@ -12,4 +13,5 @@ repository-guide
 development-testing
 pypi-release
 documentation-deployment
+defect-tracking
 ```

{operondbs-0.8.0 → operondbs-0.8.2}/docs/en/guides/phylogenetics-workflow.md

@@ -66,7 +66,7 @@ Roles are freely named by your workflow. Every product is materialized under `an
 Subfamily assignment is the point where the pipeline's shape becomes data-dependent: only after the external assign-subfamilies step reads the adopted framework tree is the number of subfamilies known. Do not hand-write one adopt manifest entry per subfamily. Have the external step emit a clean two-column assignment TSV (`unit`, `seqid` — project-specific filtering such as dropping unresolved sequences happens there), adopt that TSV as shown in step 3, then admit the units with `operon fanout`:
 
 ```bash
-# Inspect the plan first: no files, no run row.
+# Inspect the plan first: full preflight runs, but no files and no run row.
 operon fanout --assignments-file FIL_000022 \
   --source-file FIL_000012 \
   --entity-type organism --entity-id ORG_000001 \

{operondbs-0.8.0 → operondbs-0.8.2}/docs/en/guides/qc-profiles.md

@@ -40,7 +40,10 @@ TUI Config screen (`operon tui`, key `6`, QC Profiles tab): a structured form
 (description, `applies_to` checkboxes, rule rows) that validates the composed
 document, bumps the `version`, writes the file, and records the same
 content-addressed snapshot `operon evaluate` records — with snapshot history
-and restore-as-new-version built in.
+and restore-as-new-version built in. When a file has been deleted or replaced
+with an older version, the next changed save continues above the highest
+recorded version for that name. See [the TUI reference](../reference/cli-tui.md)
+for the confirmation and recovery behavior.
 
 Run the profile:
 
@@ -205,11 +208,11 @@ Grammar:
 
 - `applies_to` is a mapping with `entity_type` and `file_role` (unlike the list form of `kind: qc` profiles); it selects the target manifest files. Superseded and effectively retired entities are excluded.
 - `sources` names the hit sources the rules can reference. Each source declares:
-  - `analysis`: the analysis name whose `analysis_alignments` rows count; only rows of the latest `completed` job per target file contribute.
+  - `analysis`: the analysis name whose `analysis_alignments` rows count; only rows of the latest `completed` job per target file contribute. Other completed jobs for the same analysis+file pair are ignored, and a non-zero count is surfaced as `ignored_completed_jobs` in the run output and details.
   - `filter`: which rows count as hits at all (a list of conditions, AND-ed; empty means every row).
   - `best_by`: an ordered best-hit ranking; the first surviving row per seqid is the best hit the rules see. Each entry is a `field` with `direction: asc|desc` (default `asc`), or a `rank` map from value to rank (unmapped values sort after all mapped ones; an entry-level `default` sets their rank explicitly). Without `best_by`, rows order by `hit_rank` ascending.
 - `rules` are evaluated in order and the first match wins. A rule carries a `label` plus exactly one of: a non-empty `when` list of conditions checked against the source's best hit, `absent: true` (the source has no hit row at all for the sequence — tier U above), or `default: true` (catch-all; takes no `source`/`when`/`absent`). A sequence matched by nothing stays unlabeled.
 - Conditions — the grammar is shared between `filter` and `when` — are mappings with a `field` and an `operator`: `>=`, `<=`, `>`, `<`, `==`, `!=` (numeric, with string fallback for equality), `in`/`not_in` (with a `values` list), `between` (with `min`/`max`), `exists`, and `like` (case-insensitive SQL LIKE pattern with `%` and `_` wildcards). A condition may instead be `any: [...]` (a group whose conditions are OR-ed) or `not: {...}` (negation of a single condition); the top-level list is always AND-ed.
-- Fields resolve against the alignment columns (`subject_id`, `hit_rank`, `query_start`, `query_end`, `evalue`, `bitscore`, `percent_identity`, …), then the derived `span` field (`query_end - query_start + 1`) and `seqid` (the query id up to the first whitespace), and finally the keys of the hit row's `extra_json` — parser-specific fields such as `hit_type`, `incomplete`, `short_name`, or `i_evalue`. A missing field makes the test False for every operator (including `!=`, `not_in`, and `exists`), so only wrapping the condition in `not: {...}` makes it hold — the negation of that False is True.
+- Fields resolve against the alignment columns (`subject_id`, `hit_rank`, `query_start`, `query_end`, `evalue`, `bitscore`, `percent_identity`, …), then the derived `span` field (`query_end - query_start + 1`) and `seqid` (the query id up to the first whitespace), and finally the keys of the hit row's `extra_json` — parser-specific fields such as `hit_type`, `incomplete`, `short_name`, or `i_evalue`. A missing field never satisfies a condition in any form: the test is False for every operator (including `!=`, `not_in`, and `exists`), a `not: {...}` negation of a missing-field condition is still False, and a missing-field disjunct inside `any: [...]` counts as unsatisfied. `between` requires numeric operands and raises a validation error naming the field and value otherwise; `in`/`not_in` compare values as strings.
 
 Re-running with the same profile content and the same inputs is a no-op; editing the profile re-decides the affected sequences and audits every label change in `changes`.