OperonDBS 0.8.4__tar.gz → 0.9.0__tar.gz

{operondbs-0.8.4 → operondbs-0.9.0}/.github/workflows/publish.yml

@@ -78,6 +78,8 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v6
+        with:
+          persist-credentials: false
       - uses: actions/setup-python@v7
         with:
           python-version: "3.12"
@@ -133,11 +135,13 @@ jobs:
       name: pypi
       url: https://pypi.org/project/operondbs/
     permissions:
-      id-token: write
+      # This is a standard OIDC Keyless Authorization Statements,
+      # NOT a hardcoding token
+      id-token: write  # nosemgrep: security.hard-coded-tokens
     steps:
       - uses: actions/download-artifact@v8
         with:
           pattern: python-package-*
           path: dist/
           merge-multiple: true
-      - uses: pypa/gh-action-pypi-publish@release/v1
+      - uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2

{operondbs-0.8.4 → operondbs-0.9.0}/.github/workflows/test.yml

@@ -6,6 +6,9 @@ on:
       - master
   pull_request:
 
+permissions:
+  contents: read
+
 # A newer push supersedes an in-flight run of the same ref.
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
@@ -49,6 +52,6 @@ jobs:
           python -m pip install --upgrade pip
           python -m pip install -e '.[test]'
           python -m pytest
-      - uses: codacy/codacy-coverage-reporter-action@v1.3.0
+      - uses: codacy/codacy-coverage-reporter-action@89d6c85cfafaec52c72b6c5e8b2878d33104c699 # v1.3.0
         with:
           project-token: ${{ secrets.CODACY_PROJECT_TOKEN }}

{operondbs-0.8.4 → operondbs-0.9.0}/AGENTS.md

@@ -26,12 +26,10 @@ principle-to-implementation mapping.
 
 Current version markers (must stay consistent across code and docs):
 
-- `operon` 0.8.4 (`pyproject.toml`)
+- `operon` 0.9.0 (`pyproject.toml`)
 - database schema 2.11 (`operon/database.py`, `SCHEMA_VERSION`)
 - metadata schema 1.4 (`operon/schema.py`, `METADATA_SCHEMA_VERSION`)
 
-The project is licensed AGPL-3.0-or-later (`LICENSE` at the repo root).
-
 ## Repository layout
 
 - `operon/` — the Python package (CLI entry points: `operon/cli.py`,
@@ -56,7 +54,10 @@ The project is licensed AGPL-3.0-or-later (`LICENSE` at the repo root).
     `tests/regression/test_cython_parser_parity.py`). `measure.py` backs
     `operon qc-measure`, a project-independent measurement-only path whose
     JSON payload can be imported back through `operon import-qc` (the remote
-    built-in QC workflow).
+    built-in QC workflow; `imports.py` is the shared core of that import —
+    payload/TSV detection, manifest validation, metric insertion, QC-state
+    recomputation and the run record — and `plan_qc_import` its write-free
+    preview).
   - `operon/execution.py` — execution backends for external commands:
     `local` subprocess, `slurm` (sbatch submit + squeue poll), and `ssh`
     (Paramiko; HPC head nodes and cloud VMs, optionally through remote
@@ -68,7 +69,9 @@ The project is licensed AGPL-3.0-or-later (`LICENSE` at the repo root).
     monitor, a Decisions screen, a Config screen, a Publish screen (nav key
     `7`; release builder + selective export builder with read-only previews),
     a Coverage screen (nav key `8`; taxonomy snapshots, reference sets,
-    coverage report generation and `COV_*` report browsing), and the import
+    coverage report generation and `COV_*` report browsing, plus *Import
+    taxonomy…* / *Compile reference set…* write entries mirroring
+    `operon taxonomy import`/`operon taxonomy compile`), and the import
     dataset wizard (`operon/tui/screens/import_wizard.py`; Home button or
     global `i`, except on the Files screen where `i` stays ingest). Read
     access lives in
@@ -83,6 +86,11 @@ The project is licensed AGPL-3.0-or-later (`LICENSE` at the repo root).
     Confirm → background worker → notify + reload or inline error. Phase 3
     actions in the same module: `import_dataset` (commits wizard drafts
     through the shared single-transaction `import_wizard._commit`),
+    `ncbi_datasets` (backing the Home screen's NCBI Datasets import dialog:
+    a mandatory dry-run preflight — `dry_run` for offline inputs,
+    `plan_only` for accession-only requests — gates Confirm, and Cancel sets
+    a cooperative `cancel_event` the core records as an `interrupted`,
+    `--resume-run`-able run),
     `reserve_entity_ids`, `create_release`, `export`, `run_coverage`
     (a below-threshold coverage report returns `exit_code=1` in the result
     dict — a warning, not an exception), `preflight_backend`, `run_analysis`
@@ -110,12 +118,15 @@ The project is licensed AGPL-3.0-or-later (`LICENSE` at the repo root).
     the project-wide label summary behind the Files screen's `l` binding), which
     has no CLI reader at all. The
     Config screen (`operon/tui/screens/config.py`, nav key `6`) edits
-    `config/profiles/*.yaml` (kind `qc` and kind `sequence_classification` —
-    each kind has its own form, dispatched by the document's own kind, with the
-    classification widgets in `operon/tui/screens/config_classification.py`
-    and `actions.save_classification_profile` sharing `save_profile`'s version,
-    snapshot and rollback machinery; a profile whose conditions nest deeper
-    than one `any:`/`not:` level opens read-only) and single recipes inside
+    `config/profiles/*.yaml` (kinds `qc`, `sequence_classification` and
+    `taxonomy_coverage` — each kind has its own form, dispatched by the document's
+    own kind, with the classification widgets in `operon/tui/screens/config_classification.py`,
+    the coverage widgets in `operon/tui/screens/config_coverage.py`, and
+    `actions.save_classification_profile` / `actions.save_coverage_profile`
+    sharing `save_profile`'s version, snapshot and rollback machinery; a
+    classification profile whose conditions nest deeper than one `any:`/`not:`
+    level, or a coverage profile whose structure exceeds the flat grammar, opens
+    read-only) and single recipes inside
     `config/tools.yaml` through structured control-based forms (no free-text
     YAML): every save bumps the `version`, records the same content-addressed
     snapshot the CLI records (`qc_profiles` / `recipe_snapshots`), and
@@ -132,7 +143,9 @@ The project is licensed AGPL-3.0-or-later (`LICENSE` at the repo root).
     `config.py` (project configuration and directory layout), `files.py`
     (immutable manifest archival and verification), `profiles.py` +
     `rules.py` (versioned QC profiles and the decision engine),
-    `workflow.py` (state machine and run logs), `tools.py` (external-tool
+    `workflow.py` (state machine and run logs), `pipeline.py` (the
+    four-stage ingest → standardize → QC → evaluate runner shared by
+    `run-pipeline`'s CLI and TUI), `tools.py` (external-tool
     recipes from `config/tools.yaml`), `taxonomy.py` + `coverage.py` (frozen
     NCBI Taxonomy snapshots and coverage denominators), `release.py` +
     `export.py` (immutable releases and selective exports), `lifecycle.py`
@@ -315,7 +328,7 @@ change:
   `docs/<language>/.readthedocs.yaml`, and the registry in
   `tests/unit/test_docs_projects.py`
 
-Version markers in docs (`operon` 0.8.4, database schema 2.11, metadata
+Version markers in docs (`operon` 0.9.0, database schema 2.11, metadata
 schema 1.4) must match `pyproject.toml` and the code. Do not write the
 current values literally in Markdown sources: use the `myst_substitutions`
 references `{{ operon_version }}`, `{{ db_schema }}`, and

operondbs-0.9.0/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2026 HYLi360.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+Redistributions of source code must retain the above copyright notice, this list
+of conditions and the following disclaimer.
+
+Redistributions in binary form must reproduce the above copyright notice, this
+list of conditions and the following disclaimer in the documentation and/or
+other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

{operondbs-0.8.4 → operondbs-0.9.0}/OperonDBS.egg-info/PKG-INFO

@@ -1,9 +1,9 @@
 Metadata-Version: 2.4
 Name: OperonDBS
-Version: 0.8.4
+Version: 0.9.0
 Summary: Operon the Database System: Archive, Quality-Control, Organize, Analyze and Release Your Bio-Data
 Author-email: hyli360 <lihuanyu2003@gmail.com>
-License-Expression: AGPL-3.0-or-later
+License-Expression: BSD-2-Clause
 Project-URL: Repository, https://github.com/HYLi360/Operon
 Project-URL: Issues, https://github.com/HYLi360/Operon/issues
 Classifier: Operating System :: MacOS

{operondbs-0.8.4 → operondbs-0.9.0}/OperonDBS.egg-info/SOURCES.txt

@@ -61,6 +61,7 @@ docs/en/operations/ncbi-recovery-migration.md
 docs/en/operations/qc-performance.md
 docs/en/reference/behaviors-and-limitations.md
 docs/en/reference/cli-analysis.md
+docs/en/reference/cli-config.md
 docs/en/reference/cli-decisions-reports.md
 docs/en/reference/cli-files-qc.md
 docs/en/reference/cli-project-metadata.md
@@ -119,6 +120,7 @@ docs/zh/operations/ncbi-recovery-migration.md
 docs/zh/operations/qc-performance.md
 docs/zh/reference/behaviors-and-limitations.md
 docs/zh/reference/cli-analysis.md
+docs/zh/reference/cli-config.md
 docs/zh/reference/cli-decisions-reports.md
 docs/zh/reference/cli-files-qc.md
 docs/zh/reference/cli-project-metadata.md
@@ -154,12 +156,14 @@ operon/lifecycle.py
 operon/lineage.py
 operon/metadata_files.py
 operon/ncbi_reconcile.py
+operon/pipeline.py
 operon/profiles.py
 operon/release.py
 operon/remotes.py
 operon/reports.py
 operon/rules.py
 operon/schema.py
+operon/secrets.py
 operon/sequence_tools.py
 operon/shutdown.py
 operon/sql.py
@@ -176,6 +180,7 @@ operon/qc/__init__.py
 operon/qc/_alignment.pyx
 operon/qc/_parsers.pyx
 operon/qc/alignment.py
+operon/qc/imports.py
 operon/qc/measure.py
 operon/qc/parsers.py
 operon/tui/__init__.py
@@ -194,6 +199,7 @@ operon/tui/screens/classify.py
 operon/tui/screens/common.py
 operon/tui/screens/config.py
 operon/tui/screens/config_classification.py
+operon/tui/screens/config_coverage.py
 operon/tui/screens/coverage.py
 operon/tui/screens/decisions.py
 operon/tui/screens/derived_ops.py
@@ -205,9 +211,12 @@ operon/tui/screens/hits.py
 operon/tui/screens/home.py
 operon/tui/screens/import_wizard.py
 operon/tui/screens/labels.py
+operon/tui/screens/ncbi_datasets.py
 operon/tui/screens/publish.py
 operon/tui/screens/run_external.py
 operon/tui/screens/runs.py
+operon/tui/screens/table_import.py
+operon/tui/screens/taxonomy.py
 tests/__init__.py
 tests/helpers.py
 tests/compatibility/__init__.py
@@ -236,6 +245,7 @@ tests/unit/__init__.py
 tests/unit/test_alignment_qc.py
 tests/unit/test_classify.py
 tests/unit/test_cli_edges.py
+tests/unit/test_config_cli.py
 tests/unit/test_config_workflow_edges.py
 tests/unit/test_core_coverage_more.py
 tests/unit/test_coverage_edges.py
@@ -244,6 +254,7 @@ tests/unit/test_defect_registry.py
 tests/unit/test_defects_cli.py
 tests/unit/test_docs_projects.py
 tests/unit/test_docs_versions.py
+tests/unit/test_env_audit.py
 tests/unit/test_environment.py
 tests/unit/test_environment_capture.py
 tests/unit/test_execution.py
@@ -261,6 +272,7 @@ tests/unit/test_ncbi_coverage_gaps.py
 tests/unit/test_ncbi_edge_cases.py
 tests/unit/test_ncbi_reconcile_edges.py
 tests/unit/test_parser_edge_paths.py
+tests/unit/test_pipeline.py
 tests/unit/test_qc_and_rules.py
 tests/unit/test_qc_edges_more.py
 tests/unit/test_qc_measure.py
@@ -275,6 +287,7 @@ tests/unit/test_schema_2_10.py
 tests/unit/test_schema_2_11.py
 tests/unit/test_schema_2_9.py
 tests/unit/test_schema_and_metadata.py
+tests/unit/test_secrets.py
 tests/unit/test_sequence_tools.py
 tests/unit/test_sequences.py
 tests/unit/test_shutdown.py
@@ -287,14 +300,22 @@ tests/unit/test_tools_edges.py
 tests/unit/test_tools_execution_more.py
 tests/unit/test_tui.py
 tests/unit/test_tui_analyze.py
+tests/unit/test_tui_button_dispatch.py
 tests/unit/test_tui_cli_parity.py
 tests/unit/test_tui_config.py
 tests/unit/test_tui_derived_ops.py
 tests/unit/test_tui_flows_more.py
+tests/unit/test_tui_import_qc.py
 tests/unit/test_tui_m3_loop.py
+tests/unit/test_tui_ncbi_datasets.py
+tests/unit/test_tui_pipeline.py
 tests/unit/test_tui_publish.py
 tests/unit/test_tui_screens_more.py
 tests/unit/test_tui_splash_terminal.py
+tests/unit/test_tui_standardize.py
+tests/unit/test_tui_table_import.py
+tests/unit/test_tui_taxonomy.py
 tests/unit/test_tui_writes.py
+tests/unit/test_user_config.py
 tests/unit/test_views_release_reports_edges.py
 tests/unit/test_workflow_cli.py

{operondbs-0.8.4 → operondbs-0.9.0}/PKG-INFO

@@ -1,9 +1,9 @@
 Metadata-Version: 2.4
 Name: OperonDBS
-Version: 0.8.4
+Version: 0.9.0
 Summary: Operon the Database System: Archive, Quality-Control, Organize, Analyze and Release Your Bio-Data
 Author-email: hyli360 <lihuanyu2003@gmail.com>
-License-Expression: AGPL-3.0-or-later
+License-Expression: BSD-2-Clause
 Project-URL: Repository, https://github.com/HYLi360/Operon
 Project-URL: Issues, https://github.com/HYLi360/Operon/issues
 Classifier: Operating System :: MacOS

{operondbs-0.8.4 → operondbs-0.9.0}/docs/conf_common.py

@@ -95,7 +95,7 @@ def source_constant(module: str, name: str) -> str:
     try:
         imported = __import__(f"operon.{module}", fromlist=[name])
         return str(getattr(imported, name))
-    except Exception:
+    except (ImportError, AttributeError):
         source = (REPO_ROOT / "operon" / f"{module}.py").read_text(encoding="utf-8")
         match = re.search(rf'^{name} = "([^"]+)"', source, re.MULTILINE)
         if match is None:

{operondbs-0.8.4 → operondbs-0.9.0}/docs/en/architecture/overview.md

@@ -120,6 +120,8 @@ How the principles map to implementations:
 | `operon/profiles.py` | Loads and validates the versioned YAML profiles shared by the rule engine and taxonomy coverage |
 | `operon/metadata_files.py` | Compatibility notice for the retired live metadata TSV directory; SQLite is the only writable metadata source |
 | `operon/qc/measure.py` | Project-independent `operon qc-measure` measurement path whose JSON payload is imported back through `operon import-qc` |
+| `operon/qc/imports.py` | The shared `operon import-qc` core: JSON-payload/TSV detection, validation against the manifest, metric insertion, QC-state recomputation and the run record; `plan_qc_import` is the write-free preview the TUI shows |
+| `operon/pipeline.py` | The shared `operon run-pipeline` core: the ingest → standardize → QC → evaluate stages, `plan_pipeline`'s write-free preflight (resolved profile, entity checks, the curated-decision gate) and a `progress` callback the CLI prints |
 | `operon/classify.py` | `sequence_classification` profiles: labels per sequence from stored alignment hits into `sequence_labels` |
 | `operon/sequence_tools.py` | `extract-domains` / `select-sequences`: materializes FASTA subsets from stored alignment intervals |
 | `operon/fanout.py` | Data-derived fan-out of registered sequence files into per-unit FASTAs under `analysis/derived/` with `file_lineage` edges |

{operondbs-0.8.4 → operondbs-0.9.0}/docs/en/guides/metadata-import.md

@@ -19,6 +19,10 @@ operon import table --table organisms --file organisms.csv \
   --on-conflict update --yes
 ```
 
+The same template/preview flow is available in the TUI: the Home screen's
+*Import table* dialog generates templates and runs the mandatory preview
+before Confirm (see the reference's Import table section).
+
 Import behavior:
 
 - Only manually managed entity/accession tables are accepted. System-managed tables such as `files`, QC results, and decisions cannot be overwritten by table import.

{operondbs-0.8.4 → operondbs-0.9.0}/docs/en/guides/remote-execution.md

@@ -94,4 +94,56 @@ recipes:
 
 See [Recipe Field Reference](../reference/recipe-fields.md#slurm-resource-overrides) for the complete field list, including the `array` / `array_concurrency` job-array keys, which apply equally to local Slurm and to remote Slurm over SSH.
 
-> The automated tests for Slurm and SSH use simulated sbatch/squeue and in-memory SSH/SFTP implementations. The SSH/SFTP, remote-only analysis, and remote Slurm paths were also smoke-tested on 2026-09-04 against a Linux OpenSSH login node, a shared GPFS filesystem, and a Slurm compute node. Each deployment should still run a short local smoke task to validate its host keys, filesystem visibility, partitions, submission, cancellation, polling, and output retrieval.
+## Smoke-testing a deployment
+
+The automated tests for Slurm and SSH use simulated sbatch/squeue and in-memory SSH/SFTP implementations. The SSH/SFTP, remote-only analysis, and remote Slurm paths have also been smoke-tested against real clusters:
+
+- 2026-09-04 — Linux OpenSSH login node, shared GPFS filesystem, Slurm compute node;
+- 2026-09-20 — OpenSSH login node with `sbatch`/`squeue`/`sacct` and a `cu` partition, mirror and compute root on one shared filesystem: `evict` → remote-only `analyze` of an HMMER recipe → `pull`, over 13 MB of input.
+
+Each deployment should still run a short task of its own; the checklist, reference run and pitfalls below are what validated the 2026-09-20 deployment.
+
+### Checklist
+
+1. **Mirror reachability and host key.** `operon remotes` must list the mirror as `ok`; an unknown host is rejected until its key is in `known_hosts` or `host_key_sha256` is pinned.
+2. **One filesystem, two node classes.** `remote_root` (inherited from `storage_remote` by default) must be visible from the login node that runs `sbatch` and from the compute nodes.
+3. **Scheduler.** `sbatch`, `squeue` and `sacct` must be in the login node's non-interactive `PATH`, and the configured partition must exist.
+4. **Tools on the compute side.** Every recipe `executable` must be on the `PATH` the submitted job inherits. `operon tools-check` probes the local machine only, so it cannot validate the remote side; a binary reachable only from an interactive shell, or only from a conda environment the non-interactive shell does not activate, fails at probe or submit time.
+5. **Reference database.** Deploy it in advance at the remote target path and declare `database_checksum`; prefer a project-relative `database` so the path is mapped into the remote root (see [Databases on SSH remotes](../reference/recipe-fields.md#databases-on-ssh-remotes)).
+6. **A short recipe over one small file**, so the smoke run takes minutes rather than hours.
+
+To also exercise remote-only input consumption, configure `execution.ssh.storage_remote` as in [Remote-First Operation](remote-first.md) and evict the file before the run.
+
+### Reference smoke run
+
+```bash
+operon evict --remote mycluster --file-id FIL_000001                    # input becomes REMOTE_ONLY
+operon analyze --analysis <recipe> --entity-id <entity> --threads 24 --backend ssh
+operon analyze --analysis <recipe> --entity-id <entity> --threads 24 --backend ssh   # must report cached
+operon pull --remote mycluster --file-id FIL_000001                     # restore the local bytes
+```
+
+Then check both planes:
+
+| Where | What to verify |
+|---|---|
+| `analysis_jobs` | `launcher = [ssh:user@host]`, `status = completed`, `output_sha256`, `environment_id`, `recipe_snapshot_id`, and a `command` naming the resolved database |
+| `execution_details` | `scheduler_job_id`, `slurm_elapsed_seconds`, `host`, and the remote `script` path under `remote_root/logs/` |
+| `execution_environments` | the probe describes the **compute node** (CPU model, cpuset), not the login node |
+| remote host | `logs/<run_id>.sbatch` contains rewritten paths only — the `cd` and every project path inside `remote_root`; `<run_id>.exitcode` holds `0` |
+| both sides | the retrieved output's SHA-256 equals `sha256sum` of the remote file |
+| local `logs/` | `<run_id>.stdout.log` / `.stderr.log` are copies retrieved from the remote job |
+| `changes` / `locations` | the eviction to `REMOTE_ONLY` and the restore to `CHECKSUM_VERIFIED` are audited, and the placeholder under `.operon/placeholders/` is gone after `pull` |
+
+A second identical run must report `cached` for the file instead of submitting another job; if it resubmits, the cache identity changed (a different backend or host/root, an edited recipe, or a different resolved `database`).
+
+### Pitfalls
+
+- **A `database` outside the project root is never mapped.** Absolute values — including `~/...`, which is expanded before mapping — are passed through verbatim, so the compute side must have that exact path. A recipe that works locally with `/data/db/Pfam-A.hmm` or `~/resources/hmm/PF00010.hmm` fails remotely. A project-relative value is mapped into `remote_root` instead, which is what most recipes want.
+- **`--dry-run` validates what needs no command execution.** The local database existence check runs, and for a remote `reference` database so does the provisioning stat, so a database that exists nowhere fails the preview with the message a real run reports — `reference database not found: <path>; edit config/tools.yaml`, or `remote reference database is not provisioned at <path>` — and the command exits non-zero. A remote `mutable_cache` has nothing to verify yet, and the tool version is still reported as `not probed (backend=…)` because probing it would execute a command. Validate a deployment with one real file, not with `--dry-run`.
+- **A remote `reference` database must declare `database_checksum`.** Without it the run is rejected before anything is submitted: `remote reference databases require database_checksum so cache identity does not depend on a missing local path`.
+- **The reference database is not a manifest file.** `push`, `evict` and `pull` move ingested files only, so `operon remotes` does not count the database and a new mirror root needs it redeployed by hand.
+- **Remote-only semantics come from the backend, not from the file's status.** Any `ssh` run with a non-empty `remote_root` skips the local database existence check, requires `database_checksum`, and mixes the SSH location into the database cache identity. Results are therefore not shared between the `local` and `ssh` backends, or between two hosts/roots — a backend switch recomputes by design.
+- **The input never travels back on its own.** A locally present input is uploaded over SFTP; a `REMOTE_ONLY` input is checked against the local manifest, the remote manifest and the live remote bytes, then consumed in place. Either way the bytes stay where they are, and `pull` remains an explicit step.
+- **Tool outputs may embed the command that produced them.** HMMER's `--tblout`/`--domtblout` banner records the query and target paths, the full argv, the working directory and the date, so the same input analyzed locally and remotely yields different bytes even when the hit table is identical (observed on one file: 135 identical data lines, 5 differing banner lines, `output_sha256` `a64cc0c5…` → `1a0fc453…`). Treat `output_sha256` as host- and path-specific: a re-run on another backend overwrites the recorded output, and the older job row's hash then no longer matches the file on disk — such a row is marked `superseded` the next time it is hit. This is another reason the cache identity includes the SSH location.
+- **Tool versions are probed through the backend, so keep the two installations compatible.** `qc-measure` payloads are the one place where a version mismatch is only a warning; for `analyze` results, a remote tool of a different version changes the recorded `tool_version` and therefore the cache identity.

{operondbs-0.8.4 → operondbs-0.9.0}/docs/en/guides/remote-first.md

@@ -124,6 +124,15 @@ operon pull --remote mycluster --file-id FIL_000001
 
 `REMOTE_ONLY` files make the local database and the remote mirror root two separate backup targets; see [Backup, Migration, and Resumption](backup-migration.md).
 
+## Common pitfalls
+
+- **The reference database is not mirrored.** `push`, `evict` and `pull` move manifest files only, so a recipe `database` must be deployed at the remote target path by hand — with a `database_checksum` for a `reference` database — and redeployed whenever the mirror moves to another host or root.
+- **Keep `database` project-relative.** An absolute or `~/...` value reaches the remote command verbatim and must exist at that exact path there; a project-relative value is mapped into the remote root instead.
+- **`--dry-run` only previews.** It skips the database checks and does not probe the tool through the backend, so a deployment is validated by the first real run, not by a dry run.
+- **`analyze` hydrates nothing.** A `REMOTE_ONLY` input is consumed in place and stays `REMOTE_ONLY`; `standardize`, `release` and `export` still need local bytes, so `pull` first.
+- **Results are location-scoped.** The database identity includes the SSH host and root, so the same recipe does not share cached results between the `local` and `ssh` backends, or between two hosts.
+- **A backend switch is not just a speed setting.** Any `ssh` run with a non-empty `remote_root` requires `database_checksum` and skips the local database existence check. See [Remote Execution with Slurm and SSH](remote-execution.md#pitfalls) for the field-level rules and the rest of the list.
+
 ## Limitations
 
 - `run-external` and `analyze` are synchronous and blocking: the local CLI process holds the remote job for its whole lifetime. Interrupting the process or hitting `--timeout` cancels the remote job (`scancel` for remote Slurm, TERM/KILL to the process group for direct SSH). There is no submit-and-disconnect mode that recovers a job later.

{operondbs-0.8.4 → operondbs-0.9.0}/docs/en/guides/taxonomy-coverage.md

@@ -28,7 +28,7 @@ Import, compilation, and reporting are explicit commands. Imported source files
 
 ## Prepare a coverage profile
 
-`operon init` creates `config/profiles/coverage_viridiplantae_v1.yaml` as an example. It is a template, not a universal standard. Before production use, review the clade, exclusion rules, and thresholds, and save the result under a filename that reflects the study scope. QC and coverage profiles share the directory but are distinguished by the required `kind` field.
+`operon init` creates `config/profiles/coverage_viridiplantae_v1.yaml` as an example. It is a template, not a universal standard. Before production use, review the clade, exclusion rules, and thresholds, and save the result under a filename that reflects the study scope. QC and coverage profiles share the directory but are distinguished by the required `kind` field. The TUI's Config screen (`operon tui`, key `6`, QC Profiles tab) edits coverage profiles with the same version bump and content-addressed snapshot recording as the CLI; a profile whose structure exceeds the form opens read-only there. Hand-editing the YAML file remains fully supported.
 
 ```yaml
 kind: taxonomy_coverage
@@ -94,6 +94,10 @@ After import:
 
 Repeating an import with the same version and bytes reuses the snapshot. The same version label with different bytes is a conflict and is rejected.
 
+The TUI offers the same import on the Coverage screen (*Import taxonomy…*
+button) with the equivalent command preview; a running import cannot be
+interrupted from the TUI.
+
 ## Compile the immutable denominator
 
 ```bash
@@ -112,6 +116,11 @@ taxonomy/reference_sets/
 └── coverage_viridiplantae_v1@2026-08-01.provenance.json
 ```
 
+The TUI offers the same compilation on the Coverage screen (*Compile
+reference set…* button): the profile and the taxonomy version are picked
+from dropdowns that list only READY snapshots; a running compile cannot be
+interrupted from the TUI.
+
 Each TSV row is a target taxon with fixed columns:
 
 ```text

{operondbs-0.8.4 → operondbs-0.9.0}/docs/en/reference/behaviors-and-limitations.md

@@ -131,7 +131,7 @@ remaining bullets are intended-but-implicit semantics or accepted limitations.
 - **A control ignores a click while its press animation runs.** Textual's `Button` swallows a click while the button still carries its `-active` press effect (roughly 0.2 s), so a rapid double-click on the same button is lost; the TUI's dialogs are built around one Confirm per action, and tests wait for the effect to clear before clicking again.
 - **The Files screen's derived-artifact dialogs leave registration to adopt.** *Extract domains* and *Select sequences* write their FASTA through the core and then hand the path to the adopt dialog (`derived_from` = the source file), because neither command registers its output — the same two-step flow as the CLI. The adopt dialog's manifest mode requires a *Preview* before Confirm (the parsed item count is shown), and `fanout`'s dry run is a mandatory first step that performs the real preflight and writes nothing at all (no run row either); changing an input invalidates the preview and disables Confirm again.
 - **The classification-profile form stops at one level of `any:`/`not:`.** The Config screen's `sequence_classification` editor models flat conditions, one `any:` group and one `not:` negation — the same forms the qc editor offers for its rules. A profile that nests deeper (or whose `sources`/`rules` are not mappings) opens read-only: the reason is printed, *Save profile* is disabled, and the file is never rewritten, so hand-written structure survives every visit to the editor. Unknown keys are preserved verbatim at every level, and the composed document keeps the original key order.
-- **A profile's kind belongs to the file, not the editor.** The TUI routes a document to the qc or the classification form by the `kind` it finds on disk, and a save refuses to cross kinds (editing a `kind: qc` file with a classification document, or vice versa, raises instead of writing). `taxonomy_coverage` profiles stay hand-edited.
+- **A profile's kind belongs to the file, not the editor.** The TUI routes a document to the qc, classification or coverage form by the `kind` it finds on disk, and a save refuses to cross kinds (editing a `kind: qc` file with a classification document, or vice versa, raises instead of writing). The `taxonomy_coverage` editor models the flat coverage grammar (NCBI source, root TaxIDs, family/genus ranks, exclusion filters, per-rank thresholds); a profile whose structure exceeds the form opens read-only, and unknown keys are preserved verbatim at the document and section level.
 
 ### TimeTree
 
@@ -250,7 +250,7 @@ remaining bullets are intended-but-implicit semantics or accepted limitations.
 - **Only some symlinks are rebased.** Absolute symlinks under `standardized/` that point inside the project are rewritten to relative targets when the backup is created; relative links, links elsewhere in the tree and absolute targets outside the project keep their exact text, and there is no restore command (`backup.py`).
 - **`backup verify` rejects unexpected extra files**, not only missing or changed ones (`backup.py`).
 - **The wizard holds the write lock while hashing and copying.** Ingest happens inside one large transaction; on failure the database rolls back, buffered JSONL records are discarded, and only newly created `raw/` targets are deleted — pre-existing targets stay (`import_wizard.py`).
-- **The wizard re-hashes inputs, burns IDs and needs a user name.** Each input is hashed two to three times and the first pass runs outside the write lock; `db.next_id()` reserves and commits IDs while prompts are still open, so a cancelled wizard permanently burns IDs; the actor is `os.environ.get("USER")`, so container or cron runs record a NULL actor; and new entities are forced to `METADATA_VALIDATED` (`import_wizard.py`, `database.py`).
+- **The wizard re-hashes inputs, burns IDs and needs a user name.** Each input is hashed two to three times and the first pass runs outside the write lock; `db.next_id()` reserves and commits IDs while prompts are still open, so a cancelled wizard permanently burns IDs; the actor is the resolved audit actor (see [user configuration](cli-config.md)), so container or cron runs without `USER`, a local account entry, or `identity.actor` record a NULL actor; and new entities are forced to `METADATA_VALIDATED` (`import_wizard.py`, `database.py`).
 - **The wizard needs a TTY and accepts only regular files** (no directory artifacts). If another process consumes a shown ID in the meantime, preflight raises and the wizard must be restarted (`import_wizard.py`).
 - **Known issue: the wizard breaks when its draft outlives the database state.** Repopulating a page assigns the draft's remembered IDs to the pickers; if an entity was retired or removed while the wizard was open, the `Select` value is no longer among the freshly loaded options and `InvalidSelectValueError` is raised on the UI thread inside the worker callback, so the page never renders and no inline error appears. For the same reason a failed startup leaves navigation enabled with an empty reserved-ID map, and pressing Next with a typed organism name then raises an uncaught `KeyError`. Leaving the sequencing/annotation page clears only the enable checkbox, so stale widget text can reappear when the section is re-enabled (`import_wizard.py`, TUI).
 - **The demo project is byte-deterministic and deliberately broken.** The synthetic project uses a fixed seed (20260816), fixed contig lengths and 400×100 bp reads with hard-coded IDs (with a gap at `ANN_000002`), release `2026.08.demo` and project `PRJ_DEMO_001`; it ships a broken annotation (601-nt CDS, dangling mRNA parent), constant `'I'` read quality and an R2 that is unrelated random DNA rather than the reverse complement; its sources live in `examples/synthetic_source`, which `full` backups copy (`demo.py`, `backup.py`).

{operondbs-0.8.4 → operondbs-0.9.0}/docs/en/reference/cli-analysis.md

@@ -57,7 +57,7 @@ Outside `--dry-run`, `analyze` prints one progress line per file as processing a
 
 Slurm array submission: when the recipe's `slurm:` block sets `array: true`, the selected executor supports job arrays (the local Slurm backend, or the SSH backend with `scheduler: slurm`), and at least two files in the batch miss the cache, those files are submitted as a single Slurm job array instead of one job per file; `array_concurrency: N` throttles how many array tasks run concurrently (Slurm `%N`). In every other case the per-file submission path is used unchanged. Each file still gets its own `workflow_runs` row, with the per-task scheduler job ID recorded as `<array_id>_<task_index>`. Cache behavior is unchanged: array participation never enters the parameter fingerprint or cache identity, so cache hits and output adoption work identically with array submission on or off. See "Slurm array submission" in the [Recipe field reference](recipe-fields.md).
 
-`--dry-run` lists the plan without execution. Status values are `cached` (completed cache hit), `adoptable` (verified old output will be adopted), or `planned` (execution will run). The output column contains planned output paths, and `tool_version` contains the detected version.
+`--dry-run` lists the plan without execution. Status values are `cached` (completed cache hit), `adoptable` (verified old output will be adopted), or `planned` (execution will run). The output column contains planned output paths, and `tool_version` contains the detected version on the local backend; a non-local backend reports `not probed (backend=…)` because the probe itself would open a connection. A dry run does perform the validations that need no command execution — the recipe database's local existence, a remote reference database's provisioning (a stat over the backend), and the `database_checksum` rule — so a plan that cannot execute reports the same error a real run would, and the command exits non-zero.
 
 `--param` can set only parameters declared by the recipe. Missing required parameters, unknown parameters, repeated values, or values failing `pattern`/`choices` are configuration errors. Default `busco_lineage` usage:
 

operondbs-0.9.0/docs/en/reference/cli-config.md

@@ -0,0 +1,152 @@
+# User configuration
+
+`operon` keeps user-level settings in one optional YAML file outside the
+project. It never replaces project configuration: `project.yaml` still owns the
+storage layout, database path, resources, and execution backends, while the user
+file owns the handful of settings that belong to *you* rather than to a project.
+
+```{note}
+The file is optional. Without it, built-in defaults apply and the per-user
+identity falls back to the environment, exactly as before.
+```
+
+## Location
+
+The user configuration follows the XDG Base Directory specification:
+
+```text
+$XDG_CONFIG_HOME/operon/config.yml      # typically ~/.config/operon/config.yml
+```
+
+- `XDG_CONFIG_HOME` unset or empty → `~/.config/operon/config.yml`.
+- There is no `OPERON_CONFIG_HOME`: the path is fully determined by XDG.
+- `operon` never creates `~/.operon`, and there is no per-project user file.
+- The file is written with mode `0600`, its directory with `0700`.
+- `operon config` is project independent: it never opens `operon.sqlite` and
+  works before `operon init` and outside any project directory.
+
+## Precedence
+
+Command-line flags win over environment variables, environment variables win
+over the file, and the file wins over built-in defaults.
+
+| Setting | Flag | Environment | User file | Fallback |
+| --- | --- | --- | --- | --- |
+| Audit actor | `--actor` | `OPERON_ACTOR`, `USER`, `LOGNAME`, `USERNAME` | `identity.actor` | the local account (`getpass`), then no actor |
+| NCBI contact | `--email` | `NCBI_EMAIL` | `ncbi.email` | unset |
+| NCBI API key | `--api-key` | `NCBI_API_KEY` | never stored in the file | a stored secret (see below) |
+| Terminal graphics | — | `OPERON_SPLASH` | `ui.splash` | `auto` (detection) |
+
+Two details are worth stating explicitly:
+
+- The local account sits between the environment and the file: on a normal
+  login the actor is still `$USER`, and `identity.actor` only fills the gap in
+  containers and cron jobs where no login variable and no password entry exist.
+- Operations that require an actor (`retire --apply`, `restore --apply`, the
+  TUI lifecycle and curate dialogs) fail with a clear error instead of writing
+  a nameless audit row.
+
+## Keys
+
+| Key | Type | Default | Meaning |
+| --- | --- | --- | --- |
+| `schema_version` | integer | `1` | Document version; written by `operon config init` |
+| `identity.actor` | string | `""` | Fallback audit actor |
+| `ncbi.email` | string | `""` | NCBI contact address for API requests |
+| `ui.splash` | choice | `auto` | `auto`, `text`, `blocks`, or `kitty` |
+
+Unknown keys are preserved in the file but ignored by this version, and
+`operon config check` reports them. Keys that look like secret material
+(`…api_key`, `token`, `secret`, `password`, `credential`) are refused by
+`operon config set`: credentials belong in a secret backend.
+
+## Commands
+
+```bash
+operon config path
+operon config show [--effective] [--json]
+operon config get KEY
+operon config set KEY VALUE
+operon config unset KEY
+operon config check
+operon config init [--force]
+operon config secret list
+operon config secret set NAME
+operon config secret get NAME
+operon config secret clear NAME
+```
+
+- `path`: prints the resolved file path.
+- `show`: prints the stored document (defaults when the file does not exist).
+  It never creates the file. With `--effective` it prints the value that wins
+  for every setting, plus where that value comes from; `--json` switches the
+  output format.
+- `get`: prints one value and exits `0`; a key without a value prints nothing
+  and exits `1`, like `git config`.
+- `set` / `unset`: validate the key and the value before writing, then write
+  atomically with user-only permissions. `unset` restores the built-in default.
+- `check`: validates the YAML, warns about a group/world-readable file and
+  about unknown keys, fails on secret material, and always prints the effective
+  settings — the quickest way to see which layer wins.
+- `init`: writes the default document; refuses to overwrite without `--force`.
+
+## Secrets
+
+Credentials never live in the user file and never have to live in the
+environment. `operon` stores them through the first available system backend:
+
+1. `secret-tool` — the Secret Service (GNOME Keyring, KWallet; `libsecret-tools`).
+2. `systemd-creds --user` — encrypted credentials under
+   `$XDG_CONFIG_HOME/operon/secrets/`.
+3. `/usr/bin/security` — the macOS Keychain.
+
+All three are used through `subprocess`; no new runtime dependency is required.
+With none of them available, `operon` stops with a clear error and tells you to
+pass `--api-key` or set `NCBI_API_KEY`. The key is resolved as
+`--api-key` > `NCBI_API_KEY` > stored secret.
+
+```bash
+# Store without exposing the value in argv or shell history
+printf '%s' "$NCBI_TOKEN" | operon config secret set ncbi.api_key
+operon config secret list
+operon config secret clear ncbi.api_key
+```
+
+`secret set` reads the value from stdin, or prompts with hidden input when a
+terminal is attached; the value is never accepted as a command-line argument.
+`secret list` shows backend availability and whether each secret is set — never
+a value. `secret get` prints the value for scripts, with a warning when stdout
+is a terminal.
+
+```{note}
+`systemd-creds --user` binds a credential to the user, the machine ID and the
+kernel's boot ID. On a shared HPC home it may therefore not decrypt on another
+node: for remote execution prefer `NCBI_API_KEY` in the environment or
+`--api-key` on the command line.
+```
+
+## Examples
+
+```bash
+# Fallback identity for containers and cron jobs
+operon config set identity.actor alice
+
+# NCBI contact address, then the API key in the secret backend
+operon config set ncbi.email you@example.org
+printf '%s' "$NCBI_TOKEN" | operon config secret set ncbi.api_key
+
+# Skip graphics detection for this user without an environment variable
+operon config set ui.splash text
+
+# What wins right now, and why
+operon config show --effective
+```
+
+## Environment variables are not deprecated
+
+Every variable keeps working and keeps winning over the file. The variables
+`operon` reads directly are audited and documented in
+`tests/unit/test_env_audit.py`: an unannotated `os.environ` read fails that
+test, so any new variable has to be reviewed and documented before it ships.
+Environment capture for `environment_id` (`PROBE_ENV_VARS`) and the HTTP/SSH
+proxy variables read by `requests`/`paramiko` are unaffected.