klej 0.0.1__tar.gz

klej-0.0.1/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

klej-0.0.1/MANIFEST.in

@@ -0,0 +1 @@
+prune tests

klej-0.0.1/NOTICE

@@ -0,0 +1,2 @@
+klej
+Copyright 2026 Maciej Falkiewicz

klej-0.0.1/PKG-INFO

@@ -0,0 +1,441 @@
+Metadata-Version: 2.4
+Name: klej
+Version: 0.0.1
+Summary: The missing glue between DVC and Slurm
+Author: Maciej Falkiewicz
+License-Expression: Apache-2.0
+Project-URL: Documentation, https://github.com/macio232/klej#readme
+Project-URL: Issues, https://github.com/macio232/klej/issues
+Project-URL: Source, https://github.com/macio232/klej
+Keywords: workflow,scheduling,slurm,hpc,dvc
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Natural Language :: English
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: NOTICE
+Requires-Dist: dawgz==2.7.0
+Requires-Dist: dvc==3.66.0
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: dev
+Requires-Dist: ruff>=0.16.6; extra == "dev"
+Provides-Extra: test
+Requires-Dist: pytest>=8.0.0; extra == "test"
+Dynamic: license-file
+
+# klej
+
+`klej` is the missing glue between
+[DVC](https://dvc.org/) and [Slurm](https://slurm.schedmd.com/slurm.html). It
+turns selected DVC stages into a dependency-aware [DAWGZ](https://github.com/francois-rozet/dawgz) workflow and
+executes each stage as a separate Slurm job.
+
+The name means “glue” in Polish.
+
+## Installation
+
+`klej` requires Python 3.11 or newer.
+
+Install the pre-alpha release from PyPI as an isolated command with
+[uv](https://docs.astral.sh/uv/):
+
+```console
+uv tool install klej==0.0.1
+```
+
+Alternatively, install it into the active Python environment with pip:
+
+```console
+python -m pip install klej==0.0.1
+```
+
+To install manually from a source checkout, create a virtual environment and
+install the local project:
+
+```console
+git clone https://github.com/macio232/klej.git
+cd klej
+python -m venv .venv
+source .venv/bin/activate
+python -m pip install .
+```
+
+This is an early release intended for evaluation and research workflows, not
+production use.
+
+## Project status and safety
+
+`klej` is an early-stage research tool. It may contain bugs, and its commands,
+configuration, and Python interfaces may change without backward
+compatibility. Do not rely on it for production-critical workloads without
+independent testing.
+
+Submitting a pipeline can create many jobs that run in parallel and may
+consume substantial cluster resources or incur significant cost. Review the
+plan with `--dry-run`, understand your cluster's quotas and billing, and
+monitor submitted jobs. You are responsible for the workloads and costs
+created by your use of `klej`; use it at your own risk.
+
+## Usage
+
+Run from a DVC repository:
+
+```console
+klej repro path/to/dvc.yaml --dry-run
+klej repro path/to/dvc.yaml
+klej repro [targets ...]
+klej repro models/dvc.yaml:train --downstream
+klej repro pipelines/ -R
+klej status
+klej status WORKFLOW_ID
+klej status --failed
+klej status --finished
+klej logs WORKFLOW_ID JOB --tail 100 --follow
+klej logs --clear WORKFLOW_ID
+klej logs --clear-all --force
+klej cancel WORKFLOW_ID --dry-run
+klej lock status
+```
+
+Targets default to `./dvc.yaml` and use the same forms as `dvc repro`: stage
+names, `dvc.yaml` and `.dvc` paths, `path/dvc.yaml:stage`, foreach or matrix
+expansions, glob patterns with `--glob`, and directories with `-R`. DVC
+dependencies in other pipeline files are included automatically.
+
+Regular, foreach, matrix-expanded, frozen, import, `.dvc`, and
+checkpoint-marked stages are supported according to pinned DVC 3.66
+`dvc repro` behavior. Experiment checkpoint snapshots and resume behavior are
+not provided.
+
+The supported DVC selection and execution flags are `-f/--force`,
+`-i/--interactive`, `-s/--single-item`, `-p/--pipeline`,
+`-P/--all-pipelines`, `-R/--recursive`, `--downstream`,
+`--force-downstream`, `--allow-missing`, `--glob`, `--no-run-cache`,
+`--ignore-errors`, `--no-commit`, `--dry`, and `-v/--verbose`. `--dry-run` is
+an alias for `--dry`. `--pull` and `-q/--quiet` are not supported.
+`-k/--keep-going` is unnecessary: klej always continues independent Slurm
+branches while failed dependencies prevent their downstream jobs from
+running. `--ignore-errors` changes those dependencies from Slurm `afterok` to
+`afterany`; failures remain visible in workflow status.
+
+`-v/--verbose` reports repository and target resolution, selected and skipped
+stages, effective configuration files, profiles, dependencies, runner and
+reservation identifiers, and submitted Slurm IDs. It also enables DVC debug
+logging and klej cache/metadata/lock diagnostics in job logs. Resolved
+environment values are not printed.
+
+An explicit `--config` is one complete configuration used for every selected
+pipeline. Otherwise, repository-root `klej.yaml` is the base and a
+`klej.yaml` beside each pipeline may partially override it. Named `profiles`
+and `stages` are merged; local `commit`, `klej`, `environment`, and
+`wrapper` values replace the base values. With one explicit configuration,
+identically named stages in different pipeline files necessarily use the same
+profile mapping. Multi-pipeline runs create one cleanup job, so all effective
+pipeline configurations must resolve to the same klej-operation profile and
+environment. Align those settings or use `--config` when they differ.
+
+`klej cancel` queries Slurm and cancels only jobs that are still cancellable.
+Its output reports counts rather than printing one line per job.
+
+Dry-run output reports projected logical and physical job totals for the whole
+workflow, including commit and reservation-cleanup jobs. A normal stage is one
+logical and physical job. A Slurm stage with multiple profiles remains one
+logical job but produces one physical attempt per profile plus a success gate;
+its stage detail shows that expansion explicitly. No Slurm arrays are used.
+
+```yaml
+profiles:
+  cpu:
+    account: my-account
+    cpus: 4
+    mem: 16G
+  gpu:
+    account: my-account
+    gres: gpu:1
+    environment:
+      APPTAINER_NV: 1
+
+stages:
+  prepare: cpu
+  # Competing Slurm requests for one logical execution.
+  train: [gpu, cpu]
+
+commit: cpu
+klej: cpu
+
+environment:
+  APPTAINER_NV: 0
+
+# The wrapper receives the runner command as one final argument.
+wrapper: [./apptainer/run.sh]
+```
+
+A stage may name one profile or an ordered, non-empty list of distinct
+profiles. With two or more profiles on Slurm, klej submits one attempt per
+profile; the first allocation to atomically claim the stage runs the wrapper
+and payload. Other attempts exit with status 75 without entering either. The
+claim is permanent, so a failing winner fails the logical stage and does not
+fall back to another profile. At-most-once applies to entering the configured
+wrapper and user payload, not to allocation: multiple Slurm allocations may
+briefly start klej's coordinator. Correctness therefore requires a filesystem
+shared by the submission host and every compute node, with atomic directory
+creation and atomic file replacement. The local backend runs the stage once
+with the first profile; scalar mappings and one-item lists retain the
+single-job behavior.
+
+Alternative attempts are submitted held. klej records every physical job ID
+in the workflow plan and target reservation before releasing them. The
+winning attempt best-effort cancels its siblings; reservation cleanup retries
+active registered physical jobs other than itself, waits for Slurm's
+configured `KillWait` plus five seconds after any cancellation, and confirms
+they have left `squeue` before releasing the reservation. If the configuration
+cannot be read, cleanup warns and uses Slurm's 30-second `KillWait` default. A
+query or cancellation failure, or a job still present after the wait, fails
+cleanup and retains the reservation for inspection and explicit lock
+recovery. If submission or registration fails after `sbatch` may have run,
+klej cancels all known jobs, including cleanup, and retains the target
+reservation for explicit lock recovery. Once all physical IDs are registered,
+a later submission failure cancels every other job but preserves cleanup so it
+can release the reservation safely.
+
+Version-2 workflow records are readable from their initial planned state, even
+before any Slurm ID exists. Cancellation and submission recovery combine IDs
+from the physical plan and the DAWGZ scheduler record so a partially persisted
+submission does not leave a known job behind.
+
+The opt-in
+[`alternative-executors-smoke`](examples/alternative-executors-smoke)
+example provides a low-cost real-cluster verification pipeline. It records
+wrapper and payload entry for a successful race and an intentionally failing
+winner, with commands for inspecting logical and physical status and logs.
+Cluster policies and accounting differ, so review its profiles, quotas, and
+costs before submitting it.
+
+## Commit modes
+
+By default, `klej repro` creates one commit job after all leaf compute jobs
+finish. That job commits the selected `dvc.yaml` using the `commit` profile
+from `klej.yaml`.
+
+Three mutually exclusive flags provide alternative metadata placement:
+
+* `--commit-per-stage` runs `dvc commit` for each stage target inside its
+  compute job. The commit therefore uses the stage's profile.
+* `--separate-commit-jobs` creates a dedicated commit job after each compute
+  job. These jobs use the `commit` profile, and downstream computation waits
+  for the upstream commit job to finish.
+* `--no-dvc-metadata` executes stages without updating DVC metadata or caches.
+
+Independently, DVC-compatible `--no-commit` still updates `dvc.lock` but does
+not cache outputs or create run-cache entries.
+
+```console
+# One final commit job (default)
+klej repro path/to/dvc.yaml
+
+# Commit each target in its compute job
+klej repro path/to/dvc.yaml --commit-per-stage
+
+# Commit each target in a separate job
+klej repro path/to/dvc.yaml --separate-commit-jobs
+
+# Update dvc.lock without caching outputs
+klej repro path/to/dvc.yaml --no-commit
+
+# Do not update DVC metadata
+klej repro path/to/dvc.yaml --no-dvc-metadata
+```
+
+All commits are serialized through klej's repository-wide commit lock and
+invoke `dvc commit --force`.
+
+## Execution environment
+
+klej installs an immutable, standalone compute runner under `.klej/runners/`
+in the directory where the command is run and keeps shared coordination data
+under `.klej/locks/` there. Add `.klej/` to the consumer repository's ignore
+rules. The repository must be visible at the same path to submission and
+compute nodes. Containers need Python and the pinned DVC version, but do not
+need klej installed.
+
+## Wrapper examples
+
+Environment values can refer to submission-side variables as `$NAME` or
+`${NAME}`. A profile's environment is applied after the global environment.
+The wrapper runs from the DVC repository root and is used only with Slurm.
+
+Set `wrapper` to an executable that prepares the compute-node environment and
+runs its final argument as a shell command. For example, put one of the scripts
+below in the repository and reference it from `klej.yaml`:
+
+```yaml
+wrapper: [./wrappers/run.sh]
+```
+
+Make the script executable with `chmod +x wrappers/run.sh`. The environment
+must provide Python and the pinned DVC version.
+
+An Apptainer wrapper can bind the repository at the same path inside the
+container:
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+readonly PROJECT_DIR="$(pwd -P)"
+readonly IMAGE="${PROJECT_DIR}/containers/project.sif"
+
+[[ -f "${IMAGE}" ]] || {
+  printf 'container image not found: %s\n' "${IMAGE}" >&2
+  exit 1
+}
+
+exec apptainer exec \
+  --bind "${PROJECT_DIR}:${PROJECT_DIR}" \
+  --pwd "${PROJECT_DIR}" \
+  "${IMAGE}" bash -c "$1"
+```
+
+For a Conda environment:
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+exec conda run --no-capture-output -n my-environment bash -c "$1"
+```
+
+For a virtual environment stored in the repository:
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+source .venv/bin/activate
+exec bash -c "$1"
+```
+
+The equivalent wrapper using uv is:
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+exec uv run -- bash -c "$1"
+```
+
+To load environment modules provided by the cluster:
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+module load python/3.12 cuda/12.4
+exec bash -c "$1"
+```
+
+Replace the example environment names, image path, and module versions with
+those available on the cluster. Add `--nv` to `apptainer exec` when the
+container needs NVIDIA GPU access.
+
+## Local backend
+
+`--backend local` is intended for development and CI validation of klej's
+planning and execution path. It is not a full alternative to `dvc repro` and
+does not promise Slurm backend parity.
+
+## Examples
+
+Runnable pipelines are available in [`examples`](examples/README.md). Each
+example is an independent DVC repository with its own `dvc.yaml` and
+`klej.yaml`.
+
+## Workflow status
+
+`klej status` lists active Slurm workflows recorded in the current `.dawgz`
+directory. A workflow is active while any job is pending or running. The
+summary shows the workflow ID, DVC pipeline, and counts of pending, running,
+succeeded, failed, cancelled, and unknown jobs. Counts include compute,
+commit, and reservation-cleanup jobs. Local-backend workflows are omitted.
+
+`klej status WORKFLOW_ID` prints the aggregate state and every DAWGZ job in
+stable zero-based order. It deliberately does not truncate or page large
+foreach or matrix workflows. New compute jobs are identified by their exact
+expanded DVC target, such as `dvc.yaml:simulate@0`; internal jobs retain their
+descriptive DAWGZ name. Older records without exact-target metadata remain
+readable using their recorded names. The `START TIME` column contains Slurm's
+unmodified accounting start date-time, or `-` when it is unavailable. Rows
+with known start times are ordered from oldest to newest; missing start times
+appear last in stable DAWGZ order.
+
+For alternative resource profiles, status shows one logical row and summary
+count per stage. The winning attempt and success gate determine that row;
+losing attempts do not count as failed workloads. Cancelling a claimed stage
+is reported as cancelled unless its payload failure was already recorded or
+the claimant failed in Slurm. Use
+`klej status WORKFLOW_ID --physical` to append every attempt, gate, commit, and
+cleanup job with its raw Slurm state and exit code. Losing siblings appear as
+`superseded`. Workflow terminality and safe record clearing still account for
+all physical jobs, including rows hidden from the default logical view.
+
+`klej status --failed` lists terminal Slurm workflows containing at least one
+failed or cancelled job. A workflow containing an unknown job is not known to
+be terminal. Slurm accounting has a cluster-defined retention window: once a
+job disappears from `sacct`, klej reports it as unknown, so sufficiently old
+failed workflows may no longer appear in this listing.
+
+`klej status --finished` lists every recorded workflow known to be terminal,
+including local-backend records, with an aggregate status of `succeeded`,
+`failed`, or `cancelled`. Its `SUBMISSION` column contains the DAWGZ workflow
+submission date-time, and rows are ordered from oldest to newest. Legacy
+records without that timestamp show `-` after timestamped workflows. Pending,
+running, and unknown workflows are omitted.
+
+Slurm states are normalized to pending, running, succeeded, failed, cancelled,
+or unknown while the complete scheduler value remains visible in detailed
+status. For example, `COMPLETED+` is succeeded, `CANCELLED by 12345` is
+cancelled, and timeout, out-of-memory, and node failures are failed.
+
+## Job logs
+
+`klej logs WORKFLOW_ID JOB` writes the currently available DAWGZ log directly
+to standard output without a heading. `JOB` may be the zero-based index from
+detailed status, an exact Slurm ID, or the exact displayed target/name when it
+is unique. Internal jobs, including separate commit jobs, use their job names
+as identities. For an alternative-profile stage, its target, name, or logical
+index selects the attempt that claimed the race, including a claimant whose
+payload failed. An exact Slurm ID instead selects that physical job directly,
+so superseded attempts and internal gate, commit, or cleanup jobs remain
+inspectable. Exact Slurm IDs take precedence over logical indexes. Ambiguous
+selectors are rejected with the matching indices.
+
+Use `--tail N` to start with the last `N` lines and `--follow` to stream new
+content. Combining them follows from the tailed content; without `--tail`,
+following starts with the complete existing log. If an active job has not
+created its log yet, follow mode waits and reports that once on standard error.
+Following stops after the job becomes terminal and all available bytes have
+been emitted. Submission tracebacks recorded by DAWGZ are printed when no
+Slurm log was created. If an alternative race is unresolved, a normal request
+lists its attempt IDs, profiles, and states. Follow mode waits while an attempt
+can still claim the race and then follows the claimant. When accounting has no
+state for an attempt, follow mode checks whether it remains in the Slurm queue
+before waiting. It does not choose an attempt merely because its log appeared
+first. Detailed status and logs are not supported for local-backend workflow
+records.
+
+`klej logs --clear WORKFLOW_ID` removes a known-terminal workflow's complete
+DAWGZ record, including its logs, and removes its rows from `workflows.csv`.
+`klej logs --clear-all` does the same for every known-terminal Slurm and local
+workflow while retaining pending, running, and unknown records. Both commands
+ask for confirmation; `--force` skips that prompt but never permits deletion
+of an active or unknown workflow. For versioned plans, this safety check
+includes every physical attempt and internal gate, commit, and cleanup job,
+including jobs hidden by the logical status view. Cleanup does not cancel
+jobs, release klej locks, or delete DVC outputs.
+
+Do not run record cleanup concurrently with workflow submission. DAWGZ does
+not coordinate updates to its `workflows.csv` index with external cleanup.