OperonDBS 0.6.2__tar.gz → 0.7.0__tar.gz

{operondbs-0.6.2 → operondbs-0.7.0}/.github/workflows/publish.yml

@@ -28,7 +28,7 @@ jobs:
       - run: python -m pip install --upgrade build twine
       - run: python -m build --sdist
       - run: python -m twine check dist/*
-      - uses: actions/upload-artifact@v5
+      - uses: actions/upload-artifact@v7
         with:
           name: python-package-sdist
           path: dist/*.tar.gz
@@ -65,7 +65,7 @@ jobs:
           CIBW_TEST_COMMAND: >-
             python -c "import operon; import operon.qc_module._parsers;
             print(operon.__version__)" && operon --help
-      - uses: actions/upload-artifact@v5
+      - uses: actions/upload-artifact@v7
         with:
           name: python-package-wheel-${{ matrix.artifact }}
           path: wheelhouse/*.whl
@@ -80,7 +80,7 @@ jobs:
     permissions:
       id-token: write
     steps:
-      - uses: actions/download-artifact@v6
+      - uses: actions/download-artifact@v8
         with:
           pattern: python-package-*
           path: dist/

{operondbs-0.6.2 → operondbs-0.7.0}/AGENTS.md

@@ -187,4 +187,25 @@ change:
   `docs/*/index.md`
 
 Version markers in docs (`operon` 0.6.2, database schema 2.9, metadata
-schema 1.4) must match `pyproject.toml` and the code.
+schema 1.4) must match `pyproject.toml` and the code. Do not write the
+current values literally in Markdown sources: use the `myst_substitutions`
+references `{{ operon_version }}`, `{{ db_schema }}`, and
+`{{ metadata_schema }}`, which `docs/conf.py` resolves from the single
+sources above at build time. Substitutions expand in paragraph text only,
+never inside code spans or fenced code blocks — examples there use
+`<version>` placeholders instead. Intentional historical pins stay literal — either on the
+allowlisted era-pinned pages (`docs/*/operations/database-compatibility.md`,
+`docs/*/operations/ncbi-recovery-migration.md`) or on a line carrying an
+inline `<!-- version-pin -->` marker — and
+`tests/unit/test_docs_versions.py` fails on any other hardcoded current
+version. Only this `AGENTS.md` keeps literal current markers (it is not
+Sphinx-rendered); update the list above when bumping.
+
+## Special Note For Codex/ChatGPT
+
+To ensure security, code testing in Codex/ChatGPT runs in a sandbox by default.
+However, this causes the `test_tui.py` section to experience Textual/asyncio
+cleanup blocking during testing, resulting in a "FAIL" report due to a timeout.
+
+To run the full test suite, first exclude `test_tui.py`, then run it separately
+outside the sandbox.

{operondbs-0.6.2 → operondbs-0.7.0}/OperonDBS.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: OperonDBS
-Version: 0.6.2
+Version: 0.7.0
 Summary: Operon the Database System: Archive, Quality-Control, Organize, Analyze and Release Your Bio-Data
 Author-email: hyli360 <lihuanyu2003@gmail.com>
 License-Expression: AGPL-3.0-or-later

{operondbs-0.6.2 → operondbs-0.7.0}/OperonDBS.egg-info/SOURCES.txt

@@ -60,6 +60,7 @@ docs/en/operations/database-compatibility.md
 docs/en/operations/index.md
 docs/en/operations/ncbi-recovery-migration.md
 docs/en/operations/qc-performance.md
+docs/en/reference/behaviors-and-limitations.md
 docs/en/reference/cli-analysis.md
 docs/en/reference/cli-decisions-reports.md
 docs/en/reference/cli-files-qc.md
@@ -111,6 +112,7 @@ docs/zh/operations/database-compatibility.md
 docs/zh/operations/index.md
 docs/zh/operations/ncbi-recovery-migration.md
 docs/zh/operations/qc-performance.md
+docs/zh/reference/behaviors-and-limitations.md
 docs/zh/reference/cli-analysis.md
 docs/zh/reference/cli-decisions-reports.md
 docs/zh/reference/cli-files-qc.md
@@ -197,6 +199,7 @@ tests/unit/test_cli_edges.py
 tests/unit/test_config_workflow_edges.py
 tests/unit/test_coverage_edges.py
 tests/unit/test_database_edges_more.py
+tests/unit/test_docs_versions.py
 tests/unit/test_environment.py
 tests/unit/test_execution.py
 tests/unit/test_execution_edges.py

{operondbs-0.6.2 → operondbs-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: OperonDBS
-Version: 0.6.2
+Version: 0.7.0
 Summary: Operon the Database System: Archive, Quality-Control, Organize, Analyze and Release Your Bio-Data
 Author-email: hyli360 <lihuanyu2003@gmail.com>
 License-Expression: AGPL-3.0-or-later

{operondbs-0.6.2 → operondbs-0.7.0}/docs/conf.py

@@ -2,22 +2,60 @@
 
 from __future__ import annotations
 
+import re
 from importlib.metadata import PackageNotFoundError, version
 from pathlib import Path
 
+try:
+    import tomllib
+except ModuleNotFoundError:  # Python 3.10
+    import tomli as tomllib
+
 
 DOCS_DIR = Path(__file__).resolve().parent
+REPO_ROOT = DOCS_DIR.parent
 
 project = "Operon"
 author = "Operon contributors"
 copyright = "2026, Operon contributors"
 
-try:
-    release = version("OperonDBS")
-except PackageNotFoundError:
-    release = "0.6.2"
+
+def _package_version() -> str:
+    try:
+        return version("OperonDBS")
+    except PackageNotFoundError:
+        with (REPO_ROOT / "pyproject.toml").open("rb") as handle:
+            return tomllib.load(handle)["project"]["version"]
+
+
+def _source_constant(module: str, name: str) -> str:
+    """Read a module-level string constant, falling back to the source file."""
+
+    try:
+        imported = __import__(f"operon.{module}", fromlist=[name])
+        return str(getattr(imported, name))
+    except Exception:
+        source = (REPO_ROOT / "operon" / f"{module}.py").read_text(encoding="utf-8")
+        match = re.search(rf'^{name} = "([^"]+)"', source, re.MULTILINE)
+        if match is None:
+            raise RuntimeError(f"cannot resolve {name} from operon/{module}.py")
+        return match.group(1)
+
+
+release = _package_version()
 version = release
 
+# Markdown sources reference these as {{ operon_version }} / {{ db_schema }} /
+# {{ metadata_schema }} in paragraph text. Substitutions do not expand inside
+# code spans or fenced code blocks, so examples there use `<version>`
+# placeholders instead. Historical version mentions stay literal and are
+# guarded by tests/unit/test_docs_versions.py.
+myst_substitutions = {
+    "operon_version": release,
+    "db_schema": _source_constant("database", "SCHEMA_VERSION"),
+    "metadata_schema": _source_constant("schema", "METADATA_SCHEMA_VERSION"),
+}
+
 extensions = ["myst_parser"]
 source_suffix = {".md": "markdown"}
 root_doc = "index"
@@ -27,7 +65,7 @@ templates_path = ["_templates"]
 # resolves their relative Markdown links as Sphinx cross-references, while the
 # toctrees provide one coherent navigation hierarchy for both languages.
 myst_heading_anchors = 4
-myst_enable_extensions = ["colon_fence", "deflist", "fieldlist"]
+myst_enable_extensions = ["colon_fence", "deflist", "fieldlist", "substitution"]
 
 exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
 nitpicky = True

{operondbs-0.6.2 → operondbs-0.7.0}/docs/en/architecture/extensibility.md

@@ -2,7 +2,7 @@
 
 ## Current boundaries
 
-The built-in source adapter currently covers NCBI Datasets; sources such as ENA remain part of the future extension boundary. Taxonomy coverage currently supports only NCBI Taxonomy; GTDB and the NCBI↔GTDB crosswalk are not yet implemented. Built-in QC covers file level, reads basics, assembly structure, and annotation structure. BUSCO is natively integrated through directory output and a JSON summary parser; tools without a parser yet — QUAST, Merqury, Kraken2, CheckM2, and similar — can still be integrated through `run-external` + `import-qc`. Downstream comparative-genomics analysis is done by external workflows in `analysis/`; `operon` is responsible for data admission, provenance, and publication.
+The built-in source adapter currently covers NCBI Datasets; sources such as ENA remain part of the future extension boundary. Taxonomy coverage currently supports only NCBI Taxonomy; GTDB and the NCBI↔GTDB crosswalk are not yet implemented. Built-in QC covers file level, generic sequence basics for non-genome FASTA (CDS/protein), reads basics, assembly structure, and annotation structure. BUSCO is natively integrated through directory output and a JSON summary parser; tools without a parser yet — QUAST, Merqury, Kraken2, CheckM2, and similar — can still be integrated through `run-external` + `import-qc`. Downstream comparative-genomics analysis is done by external workflows in `analysis/`; `operon` is responsible for data admission, provenance, and publication.
 
 The contract between downstream workflows and the database is a closed loop formed by `operon export` and `operon adopt`: export materializes the selected entities by file identity into a `data/<entity_type>/<entity_id>/<filename>` layout, accompanied by `manifest.tsv` (with SHA-256 recomputed over the materialized bytes), a `qc.tsv` QC long-table snapshot, `checksums.sha256`, and `provenance.json` (the input-side manifest); after consuming this artifact set, the external workflow uses adopt to re-register derived artifacts as first-class manifest members — materialized under `analysis/adopted/<entity_id>/`, inheriting the ingest idempotency/conflict invariants, with file-to-file lineage edges recorded in `file_lineage` (the output-side manifest). Adopted products can be QC'd, evaluated, exported, released, and selected by `analyze` as inputs of downstream recipes (cascading analysis). Downstream workflows should read and write through this contract instead of reading the database directly. Orchestration of cascading workflows (dependency graphs, parallelism, retries) belongs to workflow managers such as snakemake/nextflow; `operon` is responsible for data admission, lineage, and publication, while `run-pipeline` only covers simple single-file chaining. The batch adopt manifest format is described in the [external analysis guide](../guides/external-analysis.md). Export is semantically complementary to release: release targets publication (QC-gated, immutable snapshot), while export targets analysis inputs (arbitrary selection criteria, materialized on demand).
 

{operondbs-0.6.2 → operondbs-0.7.0}/docs/en/architecture/external-analysis.md

@@ -30,9 +30,9 @@ Execution environment capture (schema 2.8, `environment.py`): all three backends
 
 A failed probe leaves the run's `environment_id` NULL without raising an error or affecting the run; rows from before 2.8 are likewise NULL.
 
-Recipe versioning and snapshots (schema 2.9): a recipe gains an optional `version:` field (a positive integer, default 1; invalid values are rejected at configuration validation). As `analyze` processes each candidate file it records the current recipe together with the verbatim spec of its referenced tool into the `recipe_snapshots` table: the snapshot document is `{"recipe": <the recipe's raw mapping>, "tool": <the referenced tool spec's raw mapping>}`, content-addressed by the SHA-256 of its canonicalized JSON and deduplicated by `UNIQUE(recipe_name, recipe_version, recipe_sha256)` — so edits to the tool definition also produce a new snapshot, and cache hits record a snapshot of the current configuration as well. `analysis_jobs.recipe_snapshot_id` points back to the exact configuration that produced the job; jobs adopted during resume inherit the original job's snapshot id (they were produced by that configuration, not today's). Inspect them with `operon recipes list / history / show`; the QC-profile counterparts recorded in `qc_profiles` are inspected with `operon profiles history / show`. Restoration via the CLI is print-only in both cases: a human copies the output back into the configuration YAML, and the CLI never rewrites files in place. The audited alternative is the TUI Config screen, whose structured editors save every change as the next version with a new snapshot and can restore any recorded snapshot into the editor (saving it creates the next version); TUI recipe saves normalize `tools.yaml` formatting and drop hand-written comments.
+Recipe versioning and snapshots (schema 2.9): a recipe gains an optional `version:` field (a positive integer, default 1; invalid values are rejected at configuration validation). As `analyze` processes each candidate file it records the current recipe together with the verbatim spec of its referenced tool into the `recipe_snapshots` table: the snapshot document is `{"recipe": <the recipe's raw mapping>, "tool": <the referenced tool spec's raw mapping>}`, content-addressed by the SHA-256 of its canonicalized JSON and deduplicated by `UNIQUE(recipe_name, recipe_version, recipe_sha256)` — so edits to the tool definition also produce a new snapshot, and cache hits record a snapshot of the current configuration as well. `analysis_jobs.recipe_snapshot_id` points back to the exact configuration that produced the job; jobs adopted during resume inherit the original job's snapshot id (they were produced by that configuration, not today's). Inspect them with `operon recipes list / history / show`; the QC-profile counterparts recorded in `qc_profiles` are inspected with `operon profiles history / show`. Restoration via the CLI is print-only in both cases: a human copies the output back into the configuration YAML, and the CLI never rewrites files in place. The audited alternative is the TUI Config screen, whose structured editors save every change as the next version with a new snapshot and can restore any recorded snapshot into the editor (saving it creates the next version); TUI recipe saves normalize `tools.yaml` formatting and drop hand-written comments. <!-- version-pin -->
 
-Run resource-usage recording (schema 2.9): `workflow_runs` gains `duration_seconds` (wall clock; previously only present in the JSONL), `avg_rss_mb` (average RSS), and `cpu_seconds` (core-seconds), and the pre-existing `max_rss_mb` column is now actually populated. Collection is per backend:
+Run resource-usage recording (schema 2.9): `workflow_runs` gains `duration_seconds` (wall clock; previously only present in the JSONL), `avg_rss_mb` (average RSS), and `cpu_seconds` (core-seconds), and the pre-existing `max_rss_mb` column is now actually populated. Collection is per backend: <!-- version-pin -->
 
 - `local`: a sampling thread polls `VmRSS` in `/proc/<pid>/status` when procfs is available and otherwise uses the POSIX `ps` RSS field (including on macOS) for peak and average; core-seconds come from the `getrusage(RUSAGE_CHILDREN)` delta across the run;
 - `slurm`: after the job finishes, `sacct` is queried with extended fields (`MaxRSS`/`AveRSS`/`Elapsed`/`TotalCPU`); remote Slurm (`ssh` with `scheduler: slurm`) follows the same path;

{operondbs-0.6.2 → operondbs-0.7.0}/docs/en/architecture/metadata-and-data-model.md

@@ -50,7 +50,7 @@ Identity and relationship policy:
 - Records without a BioSample use an assembly-specific sample;
 - Annotation identity includes source accession, provider, version, and release date, with files automatically assigned to the corresponding `ANN_`; pre-2.6 rows are continued with strictly identical metadata, avoiding duplicate assignment when the provider is not `NCBI *`.
 
-Before writing metadata, the adapter computes SHA-256 for the files to be archived and checks both in-package conflicts for the same entity/role and existing manifest conflicts. Alternate genomes/reports from paired sources use controlled roles with `_genbank`/`_refseq` suffixes, so bytes from different sources can coexist without relaxing the no-overwrite constraint on the same entity and role. Original reports/ZIPs are stored by SHA-256 under `raw/metadata/ncbi_datasets/`; import summaries are written to `changes` and the workflow provenance. On a formal import into an old project, adapter-owned fields and source-file roles are merged in, and the metadata schema is upgraded to 1.4; custom fields are preserved, and dry runs use only the in-memory upgraded schema.
+Before writing metadata, the adapter computes SHA-256 for the files to be archived and checks both in-package conflicts for the same entity/role and existing manifest conflicts. Alternate genomes/reports from paired sources use controlled roles with `_genbank`/`_refseq` suffixes, so bytes from different sources can coexist without relaxing the no-overwrite constraint on the same entity and role. Original reports/ZIPs are stored by SHA-256 under `raw/metadata/ncbi_datasets/`; import summaries are written to `changes` and the workflow provenance. On a formal import into an old project, adapter-owned fields and source-file roles are merged in, and the metadata schema is upgraded to {{ metadata_schema }}; custom fields are preserved, and dry runs use only the in-memory upgraded schema.
 
 An adapter run writes a `running` workflow before processing begins; each accession's state is kept in `adapter_run_items`. Failed or interrupted runs keep their state, and a resumed run uses a new run ID with `resumes_run_id`; a request whose SHA-256 does not match is refused. Field-level before/after values of metadata upserts are linked to the concrete run through `changes.workflow_run_id`. Anomalies from the old adapter are handled by an explicit `ncbi-reconcile` that generates and applies a compensation plan, preserving all old rows and files through `entity_supersessions`.
 

{operondbs-0.6.2 → operondbs-0.7.0}/docs/en/architecture/overview.md

@@ -1,6 +1,6 @@
 # Architecture overview
 
-This document corresponds to `operon` 0.6.2, internal database schema 2.9, and metadata schema 1.4.
+This document corresponds to `operon` {{ operon_version }}, internal database schema {{ db_schema }}, and metadata schema {{ metadata_schema }}.
 
 ## Design goals
 
@@ -22,7 +22,7 @@ How the principles map to implementations:
 | Raw immutable, standardized derived | Atomic ingest + `ConflictError` + independent copies by default |
 | Filenames contain only stable ID/role/format/compression | `canonical_filename()` |
 | Paths are not file identity | `files.file_id + sha256 + size_bytes` |
-| Layered QC | `file_integrity/reads_basic/assembly_basic/annotation_basic` |
+| Layered QC | `file_integrity/reads_basic/sequence_basic/assembly_basic/annotation_basic` |
 | Measurement separated from decision | `qc_results` long table + YAML profile rule engine |
 | Taxonomy coverage does not drift with upstream upgrades | NCBI taxonomy snapshot + compiled reference-set TSV + SHA-256 |
 | Automated state machine, explicit failures, idempotent resume | `entity_state` + strict transitions + atomic operations |
@@ -90,7 +90,7 @@ How the principles map to implementations:
 | `operon/cli.py` | argparse command parsing, dispatch, human-readable output |
 | `operon/config.py` | Reads `project.yaml`, locates the project root, generates the directory structure |
 | `operon/schema.py` | Built-in metadata field definitions, type validation and normalization, derived TSV output |
-| `operon/database.py` | SQLite DDL, WAL/foreign keys/indexes, development-time compatibility migrations and incremental schema 2.2–2.9 migrations, transactions, read-only queries |
+| `operon/database.py` | SQLite DDL, WAL/foreign keys/indexes, development-time compatibility migrations and incremental schema 2.2–{{ db_schema }} migrations, transactions, read-only queries |
 | `operon/files.py` | File format/compression detection, atomic archiving, idempotent ingest, checksum verification, standardized views |
 | `operon/lifecycle.py` | Retire/restore plans, append-only lifecycle events, hierarchical propagation, and the current retired list |
 | `operon/import_wizard.py` | English questionary import wizard, draft summary review, non-linear section editing, preflight and commit |
@@ -115,12 +115,12 @@ How the principles map to implementations:
 
 ## Project directory structure
 
-`operon init` creates the following directories and files. The SQLite database is not created at init time, but on the first command that needs it.
+`operon init` creates the following directories and files. The SQLite database is created eagerly by `operon init`; two entries appear only lazily when first needed (`logs/workflow.jsonl` on the first workflow run, `.operon/placeholders/` on the first remote-evict/pull pointer write).
 
 ```text
 project/
 ├── project.yaml              # project config: paths, default QC profile, resource parameters
-├── operon.sqlite           # file-based database (created on first command use)
+├── operon.sqlite           # file-based database (created by operon init)
 ├── config/
 │   ├── schemas.yaml          # metadata field contract (types/required/allowed values/regex)
 │   ├── tools.yaml            # external analysis programs (BLAST/HMMER/BUSCO, artifact types)
@@ -128,6 +128,7 @@ project/
 │       ├── file_integrity_v1.yaml
 │       ├── assembly_production_v1.yaml
 │       ├── annotation_release_v1.yaml
+│       ├── annotation_busco_viridiplantae_odb12_v1.yaml
 │       ├── reads_qc_v1.yaml
 │       └── coverage_viridiplantae_v1.yaml
 ├── metadata/                 # legacy layout compatibility note; no longer a read/write data source
@@ -137,8 +138,9 @@ project/
 ├── analysis/                 # analysis workspace (external tool output, downstream analysis)
 ├── reports/                  # decisions, summary exports, and coverage reports
 ├── taxonomy/reference_sets/  # compiled immutable family/genus denominators and provenance
-├── logs/workflow.jsonl       # machine-readable workflow log
+├── logs/workflow.jsonl       # machine-readable workflow log (created on the first workflow run)
 ├── .operon/placeholders/     # small, non-authoritative pointers for REMOTE_ONLY files
+│                             #   (created on the first remote evict/pull)
 └── releases/                 # immutable dataset release snapshots
 ```
 

{operondbs-0.6.2 → operondbs-0.7.0}/docs/en/architecture/qc-and-rules.md

@@ -8,6 +8,7 @@ Built-in QC loads the Cython streaming parsers by default and requires them; the
 |---|---|---|
 | `file_integrity` | All files | `file_exists`, `size_bytes`, `sha256_match`, `parseable` |
 | `assembly_basic` | genome FASTA | `total_length`, `contig_n50/n90`, `contig_l50/l90`, `gc_percent`, `n_percent` (strictly N only), `gap_count`/`gap_percent` (runs and fraction of alignment gap characters `-`), `ambiguous_base_percent`, duplicate seqids/complete headers, circular/empty sequences |
+| `sequence_basic` | other FASTA (e.g. standalone CDS or protein FASTA) | `sequence_count`, `total_length`, `empty_sequence_count`, `duplicate_sequence_id_count` |
 | `reads_basic` | FASTQ | `read_count`, `total_bases`, `q20_percent`, `q30_percent`, `gc_percent`, `duplicate_percent`, sampling count/strategy, `overrepresented_sequence_count`, read length N50, R1/R2 pairing |
 | `annotation_basic` | GFF3 (+ assembly FASTA/protein FASTA) | gene/mRNA/CDS counts, CDS triplet ratio, ID/Parent integrity, coordinate errors, seqid matching, protein duplicate IDs, X ratio, internal stop codons |
 

{operondbs-0.6.2 → operondbs-0.7.0}/docs/en/contributor/application-release.md

@@ -22,13 +22,13 @@ Release content lands in a versioned directory:
 A Linux build machine additionally needs the system command `patchelf`; it is a build-time tool for cx_Freeze's ELF dependency handling, not a Python runtime dependency of `operon`. When it is missing, cx_Freeze stops right at the `build_exe` stage.
 
 ```text
-build/release/v0.6.2/
+build/release/v<version>/
 ├── operon                  # command-line executable; operon.exe on Windows
 ├── lib/                    # Python runtime, the operon package, and third-party dependencies
 ├── LICENSE                 # Operon's own license (AGPL-3.0-or-later)
 ├── licenses/               # THIRD_PARTY_NOTICES.md and full license texts of third-party dependencies
 ├── source/
-│   └── operondbs-0.6.2.tar.gz # complete project source sdist corresponding to this binary
+│   └── operondbs-<version>.tar.gz # complete project source sdist corresponding to this binary
 ├── frozen_application_license.txt  # license of the frozen bootstrap code automatically included by cx_Freeze
 └── share/doc/operon/
     ├── README.md           # English project overview

{operondbs-0.6.2 → operondbs-0.7.0}/docs/en/contributor/development-testing.md

@@ -19,6 +19,14 @@ sphinx-build -W --keep-going -b html docs docs/_build/html
 The pytest suite is organized into four categories — `unit`, `integration`, `regression`, `compatibility` — covering: Python 3.10 syntax and runtime gates, schema validation and controlled vocabularies, metadata round-trips and transaction rollback, stable IDs, default copy isolation, query read-only constraints, file-aware QC identity, profile/decision history, gzip FASTA recognition, assembly/annotation QC, rule decisions, idempotent ingest and conflict protection, checksum-tamper detection, the demo end-to-end pipeline and release verification, the NCBI Datasets adapter, wrapped BLAST/HMMER/BUSCO execution, directory artifacts, JSON summaries, conda run prefix parsing, cache hits/forced re-runs, result write-back, and input-tamper rejection.
 The taxonomy coverage integration tests additionally cover taxonomy source-package identity conflicts, profile type/content conflicts, exclusion rules, secondary TaxIDs, denominator/report idempotence, and that active metadata modifications do not affect the release-frozen scope.
 
+## Special Note For Codex/ChatGPT
+
+To ensure security, code testing in Codex/ChatGPT runs in a sandbox by default. However, this causes the `test_tui.py` section to experience Textual/asyncio cleanup blocking during testing, resulting in a "FAIL" report due to a timeout.
+
+To run the full test suite, first exclude `test_tui.py`, then run it separately outside the sandbox.
+
+This information has also been updated in AGENTS.md.
+
 ## Documentation synchronization
 
 When changing the CLI, configuration fields, behavior, or storage layout, update the Chinese and English documentation in the same change:
@@ -31,4 +39,10 @@ When changing the CLI, configuration fields, behavior, or storage layout, update
 | `tools.yaml` recipes, placeholders, or parsers | `docs/*/reference/recipe-*.md` |
 | Migrations, performance diagnostics, or compatibility boundaries | `docs/*/operations/` |
 
-Software versions, database schema versions, and metadata schema versions stated in the documentation must stay consistent with `pyproject.toml` and the code.
+Software versions, database schema versions, and metadata schema versions stated in the documentation must stay consistent with `pyproject.toml` and the code. Write current version markers in the Markdown sources as substitutions:
+
+```text
+{{ operon_version }}  {{ db_schema }}  {{ metadata_schema }}
+```
+
+`docs/conf.py` resolves them from `pyproject.toml` and the code constants at build time. Substitutions expand in paragraph text only, not inside code spans or fenced code blocks; examples there use `<version>` placeholders instead. Intentional historical pins stay literal: they either live on the allowlisted era-pinned pages under `docs/*/operations/` or carry an inline `<!-- version-pin -->` marker. `tests/unit/test_docs_versions.py` enforces the rule.

{operondbs-0.6.2 → operondbs-0.7.0}/docs/en/contributor/pypi-release.md

@@ -31,7 +31,11 @@ manual approval before the final upload.
 
 ## Release procedure
 
-1. Update `[project].version` and every documented version marker together.
+1. Update `[project].version`, plus `SCHEMA_VERSION` / `METADATA_SCHEMA_VERSION`
+   in the code when they change. Documentation version markers render from
+   these single sources through `myst_substitutions` in `docs/conf.py`, so no
+   manual sweep is needed; `tests/unit/test_docs_versions.py` rejects
+   hardcoded current versions in the Markdown sources.
 2. Run the full pytest suite and strict documentation build.
 3. Commit the release state and create tag `v<project.version>` on that exact
    commit. Never reuse a tag that points to older package metadata.

{operondbs-0.6.2 → operondbs-0.7.0}/docs/en/getting-started/first-project.md

@@ -18,7 +18,7 @@ metadata/            Legacy-layout compatibility note
 raw/ standardized/ qc/ analysis/ reports/ logs/ releases/ taxonomy/
 ```
 
-`operon.sqlite` is created the first time a command needs the database.
+`operon init` also creates an empty `operon.sqlite` immediately, together with the directory tree.
 
 > The global `--project` option must appear before the subcommand. It can be omitted inside the project root. Outside the project, use `operon --project /path/to/my-genome-project <subcommand>`.
 

{operondbs-0.6.2 → operondbs-0.7.0}/docs/en/getting-started/installation.md

@@ -51,11 +51,13 @@ Verify the installation:
 
 ```bash
 operon --version
-# Expected: operon 0.6.2
 
 operon --help
 ```
 
+The first command prints `operon` followed by the installed version —
+{{ operon_version }} for the release this documentation matches.
+
 To build a standalone cx_Freeze application, install the build extra and use the unified release entry point:
 
 ```bash

{operondbs-0.6.2 → operondbs-0.7.0}/docs/en/guides/backup-migration.md

@@ -2,7 +2,7 @@
 
 ## Back up and migrate a project
 
-Use `backup` to create a consistent SQLite snapshot. Do not copy database files directly while the database may be active:
+Use `backup` to create a consistent SQLite snapshot. Do not copy database files directly while the database may be active. The `--output` directory must be outside the project root and must not exist yet; `backup create` refuses otherwise:
 
 ```bash
 # Configuration, SQLite, audit records, and workflow logs
@@ -17,8 +17,12 @@ operon backup create --output /backups/my-project-full --scope full
 operon backup verify --input /backups/my-project-full
 ```
 
+Note the scope boundaries: `results` excludes `raw/` and `standardized/` (the bytes you usually cannot regenerate), so it is not a restorable substitute for `full`; only `full` can restore data files.
+
 `backup verify` validates an exact snapshot. In addition to checking size and SHA-256 for files listed in the manifest, it rejects extra files in the backup directory. Keep notes, temporary files, and recovery records outside the backup directory.
 
+New backups use manifest format 2; format 1 backups remain verifiable. Symbolic links are recorded and verified by their target text, including broken links and directory links, without following their targets. Absolute links inside the standardized view that point into the project are rebased to relative paths within the backup, so a full backup can be moved independently. Links inside archived directory artifacts retain their exact text to preserve artifact identity. External link targets are not backed up; restoring such a link does not restore its external referent.
+
 With `REMOTE_ONLY` files, a local backup must include the SQLite database containing `file_locations`. Back up the remote mirror root independently, including `operon-manifest.json` and actual objects. Placeholder files are not recovery evidence. Safe hydration requires both local `files` identity and the remote manifest/bytes.
 
 `report metadata` is not a backup. It exports metadata/manifest TSV files for browsing and exchange, but does not include complete QC, decisions, changes, workflows, remote locations, or migration state.
@@ -55,6 +59,6 @@ Core steps are idempotent:
 - `release` rejects an existing version directory rather than overwriting it.
 - `taxonomy compile` reuses identical profile/taxonomy/TSV input and rejects different content under the same identity.
 - `report coverage` validates and reuses an old report when input membership, profile, and reference-set identity match.
-- After Ctrl+C/SIGTERM, `analyze` marks the current job `interrupted` and removes partial outputs. On rerun, completed files use the cache; an old result with unchanged input and verified output is adopted (`adopted`); only unfinished work is recomputed.
+- After Ctrl+C/SIGTERM, `analyze` marks the current job `interrupted` and removes partial outputs (`--keep-partial` preserves them for debugging). On rerun, completed files use the cache; an old result with unchanged input and verified output is adopted (`adopted`); only unfinished work is recomputed.
 
 Rerun the same command to continue from the interruption. Use `status` to inspect each entity's current state.

{operondbs-0.6.2 → operondbs-0.7.0}/docs/en/guides/curation-lifecycle.md

@@ -104,4 +104,15 @@ Restoration is the strict inverse operation: it appends `RESTORE` and points bac
 
 For databases older than schema 2.7, run `operon migrate` first.
 
+## Force a state transition manually
+
+`operon set-state` performs an audited manual state change when a transition is needed that the automatic workflow does not produce (for example, recovering a stuck entity):
+
+```bash
+operon set-state --entity-type assembly --entity-id ASM_000001 --state QC_COMPLETE \
+  --message "Manual review confirmed metrics are complete" --force
+```
+
+The normal path enforces the legal transition table and appends a `changes` audit row with the message; `--force` bypasses the transition check for manual recovery while the audit row keeps the action traceable. Note that `RELEASED` is a terminal state: entities published in a release cannot leave it without `--force`, and setting a state that equals the current state is a silent no-op. Prefer `curate` (for decisions) or rerunning the affected step (for provenance) whenever possible.
+
 There is currently no `purge` command. Do not use manual SQL, `rm`, or remote-object deletion as a substitute. Physical deletion requires separately designed retention periods, release/remote reference protection, a restoration window, and irreversible confirmation. Until then, auditable retirement and restoration are the supported safe-disposal path.

{operondbs-0.6.2 → operondbs-0.7.0}/docs/en/guides/external-analysis.md

@@ -92,6 +92,8 @@ Preview selection and cache status:
 operon analyze --analysis blastn_nt --dry-run
 ```
 
+Useful batch controls: `--limit N` processes only the first N matching files (in `file_id` order), and `--threads` overrides the recipe default.
+
 Inspect synchronized results:
 
 ```bash
@@ -210,6 +212,7 @@ operon run-external \
 ```
 
 - `--command` is parsed with shell-style quoting but is not run through a shell.
+- `--tool` records a tool version probed from `config/tools.yaml`; `--input` declares an input file/directory that is hashed for provenance (repeatable); `--threads`, `--cwd`, `--timeout`, and `--backend` (default `local`; also `slurm` or `ssh`) control execution.
 - stdout and stderr are saved to `logs/<WF_ID>.stdout.log` and `.stderr.log`.
 - Run records are written to `logs/workflow.jsonl` and `workflow_runs`.
 - The run is `completed` only when the exit code is 0 and every `--expected-output` exists and is non-empty; otherwise it is `failed` and the command exits non-zero.
@@ -255,5 +258,5 @@ operon adopt --from-manifest adopt_manifest.json
 ```
 
 - Each item requires `path`, `entity_type`, `entity_id`, `role`, and `derived_from` (at least one already-registered file_id); relative paths resolve from the project root.
-- Artifacts are materialized under `analysis/adopted/<entity_id>/`; same entity and role with identical bytes is reused idempotently, different bytes raise `ConflictError`; if any item is invalid (e.g. an unregistered `derived_from` or a retired entity), nothing in the batch is registered.
+- Artifacts are materialized under `analysis/adopted/<entity_id>/`; same entity and role with identical bytes is reused idempotently, different bytes raise `ConflictError`. The whole batch is preflighted, then registered in one transaction. A failure before commit rolls back metadata, lineage, state and workflow rows and removes newly created artifacts; existing files are preserved. Resolve conflicting occupied targets explicitly before retrying. Completed JSONL records are written only after commit.
 - Roles are freely named by the workflow; lineage edges are written to the `file_lineage` table and can be audited with `operon query`.

{operondbs-0.6.2 → operondbs-0.7.0}/docs/en/guides/metadata-import.md

@@ -25,7 +25,7 @@ Import behavior:
 - Schema, controlled-vocabulary, and foreign-key validation run before the row-by-row `insert/update/unchanged` preview.
 - `--on-conflict error` rejects existing rows, `skip` skips them, and `update` updates fields with per-field audit records.
 - Deletes and full-snapshot replacement are not supported. If any write fails, the transaction for the table is rolled back.
-- For XLSX, the first `data` worksheet is imported; the template's second `schema` worksheet is read-only documentation.
+- For XLSX, the first worksheet in workbook order is imported whatever its name (templates generated by `--template` name it `data` and add a read-only `schema` documentation worksheet).
 
 CSV example:
 

{operondbs-0.6.2 → operondbs-0.7.0}/docs/en/guides/ncbi-datasets.md

@@ -12,7 +12,7 @@ operon ncbi-datasets \
 
 The output includes the number of organism/sample/assembly/annotation IDs to create in `new_ids` and the rows to upsert in `metadata_rows`. A dry run does not copy input, write the database, or create logs.
 
-If the project still uses an old metadata schema, a formal import preserves custom fields, adds the fields and paired-source file roles required by the adapter, and upgrades the schema to 1.4. A dry run does not modify the schema.
+If the project still uses an old metadata schema, a formal import preserves custom fields, adds the fields and paired-source file roles required by the adapter, and upgrades the schema to {{ metadata_schema }}. A dry run does not modify the schema.
 
 After review, remove `--dry-run`:
 

{operondbs-0.6.2 → operondbs-0.7.0}/docs/en/guides/qc-profiles.md

@@ -33,7 +33,7 @@ warnings:
     code: HIGH_BUSCO_DUPLICATION
 ```
 
-Supported operators are `>=`, `<=`, `>`, `<`, `==`, `!=`, `between` (requires `min` and `max`), `in`, `not_in` (requires `values`), and `exists`.
+Supported operators are `>=`, `<=`, `>`, `<`, `==`, `!=`, `between` (requires `min` and `max`), `in`, `not_in` (requires `values`), and `exists`. Comparisons are inclusive at the boundaries, and `in`/`not_in` compare metric values as strings. Note two profile-validation gaps: a `between` rule missing `min`/`max`, or an `in`/`not_in` rule missing `values`, is not rejected when the profile loads — the former surfaces as a Python traceback at evaluation time, and the latter silently evaluates against an empty value set.
 
 Hand-editing the YAML file is fully supported. The audited alternative is the
 TUI Config screen (`operon tui`, key `6`, QC Profiles tab): a structured form

{operondbs-0.6.2 → operondbs-0.7.0}/docs/en/guides/remote-storage.md

@@ -18,6 +18,7 @@ remotes:
     # Alternatively pin an administrator-provided fingerprint:
     # host_key_sha256: SHA256:base64...
     insecure_accept_unknown_host: false
+    connect_timeout: 30            # seconds; also bounds the remote manifest lock wait
 ```
 
 Paramiko is included in the standard `OperonDBS` installation.
@@ -54,7 +55,7 @@ The remote model preserves the raw-file invariants:
 - Remote relative paths must remain safely under the remote root. By default, `pull` checks every record against local SQLite `file_id + relative_path + sha256 + size_bytes`; the remote manifest cannot rewrite local identity.
 - Every transfer writes workflow provenance (`push:<name>` or `pull:<name>`), and successful locations are recorded in `file_locations`.
 - A failed item does not stop the rest of a push/pull/evict batch. Every item receives a result, and the command exits with code 1 if any item has `error`.
-- After `pull` restores a missing local file, `files.status` returns to `CHECKSUM_VERIFIED` and the change is audited in `changes`.
+- After `pull` restores a missing local file, `files.status` returns to `CHECKSUM_VERIFIED` and the change is audited in `changes`. A file that was already `STANDARDIZED` before eviction keeps the `STANDARDIZED` status after restore.
 
 ## Keep the control plane local and large files remote
 
@@ -95,6 +96,8 @@ execution:
 
 `evict` explicitly deletes local bytes; without `--file-id`, it processes every manifest object. It first validates local identity, remote manifest identity, and actual remote SHA-256/tree hash. The state change is written to `changes`. `standardize` and `release` require local bytes, so run `pull` first. External `analyze` can consume `REMOTE_ONLY` input directly.
 
+Eviction writes a small placeholder pointer file under `.operon/placeholders/<file_id>.json` (deleted again when `pull` restores the bytes). The first remote-only status also extends `config/schemas.yaml` with the `REMOTE_ONLY` file status and bumps its `schema_version` to 1.2 — the file is rewritten with normalized formatting, so hand-written comments in it are dropped.
+
 When a local object is missing, `verify` checks the remote in real time rather than treating `file_locations.status=AVAILABLE` as permanent proof. A deleted or damaged remote object returns `MISSING` and updates the cache. An unreachable SSH host returns `REMOTE_UNVERIFIED` and exit code 1 while preserving the last persistent state, so a network failure is not misclassified as data loss.
 
 Remote files can also be archived directly from URLs:

{operondbs-0.6.2 → operondbs-0.7.0}/docs/en/guides/troubleshooting.md

@@ -18,3 +18,7 @@ Recommended actions:
 | `CHECKSUM_FAILED` | Stop QC. Determine whether the file was modified and restore it from the original source. |
 | `QC_FAILED` | Inspect files with `parseable=0` in `operon report qc`, then use `operon workflow list --step qc --status failed` and `operon workflow show WF_ID` for the recorded error and execution details. |
 | Format parsing failure | Check with an external validator such as `seqkit stats` or a GFF3 validator. Archive the repaired file as a new version; do not overwrite raw data. |
+
+## Further reading
+
+For implicit semantics, edge cases, and known issues that are not defects in a single workflow — such as how missing metrics are treated, multi-file QC state semantics, or exit-code conventions — see [Implicit Behaviors, Edge Cases, and Known Issues](../reference/behaviors-and-limitations.md).

{operondbs-0.6.2 → operondbs-0.7.0}/docs/en/index.md

@@ -2,7 +2,7 @@
 
 Operon is a file-backed database for large-scale genomic data. It supports metadata management, immutable file archiving, quality control (QC), rule-based decisions, external analysis, remote storage and execution, and versioned dataset releases.
 
-This documentation matches `operon` 0.6.2, database schema 2.9, and metadata schema 1.4. The Chinese and English documentation use the same directory structure.
+This documentation matches `operon` {{ operon_version }}, database schema {{ db_schema }}, and metadata schema {{ metadata_schema }}. The Chinese and English documentation use the same directory structure.
 
 ## Reading paths
 

{operondbs-0.6.2 → operondbs-0.7.0}/docs/en/operations/database-compatibility.md

@@ -19,7 +19,7 @@ File: `operon/database.py`
 
 `Database._migrate_remote_schema_2_2()` is also not part of the "development-era v1 compatibility layer" above. It upgrades a 2.1 database to 2.2 purely additively: adding `executor`, `scheduler_job_id`, and `execution_details` to `workflow_runs`, and creating `file_locations`. It must be kept as long as opening 2.1 projects is supported; if that support ever ends, it should be replaced through the formal database migration policy, not deleted together with `_migrate_pre_1_0_schema()`. The corresponding test is `test_schema_2_2_adds_remote_location_and_executor_provenance`.
 
-`Database._migrate_taxonomy_schema_2_3()` is likewise a purely additive migration needed by current functionality, not part of `_migrate_pre_1_0_schema()`: for 2.2 projects it creates `taxonomy_snapshots`, `taxonomy_nodes`, `taxonomy_aliases`, `taxonomy_reference_sets`, `coverage_reports`, and `coverage_report_metrics` plus related indexes, without modifying existing business rows. It must be kept as long as opening 2.2 projects is supported. The corresponding regression test is `test_schema_2_3_adds_taxonomy_coverage_history`.
+`Database._migrate_taxonomy_schema_2_3()` is likewise a purely additive migration needed by current functionality, not part of `_migrate_pre_1_0_schema()`: for 2.2 projects it creates `taxonomy_snapshots`, `taxonomy_nodes`, `taxonomy_aliases`, `taxonomy_reference_sets`, `coverage_reports`, and `coverage_report_metrics` plus related indexes, without modifying existing business rows. It must be kept as long as opening 2.2 projects is supported. The corresponding regression test is `test_schema_2_3_adds_taxonomy_and_coverage_history`.
 
 `Database._migrate_source_schema_2_4()` is another purely additive migration needed by current functionality: for 2.3 projects it creates `data_sources` and `source_links`, storing normalized external databases/repositories, citations, licenses, and their associated objects. Non-INSDC sources must contain both citation and License; source content is deduplicated by SHA-256 identity. It must be kept as long as opening 2.3 projects is supported. The corresponding regression test is `test_schema_2_4_adds_normalized_source_provenance`.
 

{operondbs-0.6.2 → operondbs-0.7.0}/docs/en/overview.md

@@ -20,6 +20,8 @@ Operon manages data admission, identity verification, provenance, rule evaluatio
 
 The current source adapter supports NCBI Datasets. Taxonomy coverage currently supports NCBI Taxonomy only; GTDB and NCBI↔GTDB crosswalks are extension work.
 
+Implicit semantics, edge cases, and known issues that are not covered by the task-facing pages are catalogued in [Implicit Behaviors, Edge Cases, and Known Issues](reference/behaviors-and-limitations.md).
+
 ## Recommended workflow
 
 ```text