data-annotations 2.3.0__tar.gz → 2.4.0__tar.gz

{data_annotations-2.3.0 → data_annotations-2.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: data-annotations
-Version: 2.3.0
+Version: 2.4.0
 Summary: Annotate generated data artifacts
 Keywords: annotations,data,metadata,provenance,reproducibility
 Author: Rodrigo C.  G.  Pena
@@ -18,6 +18,7 @@ Classifier: Programming Language :: Python :: 3.14
 Classifier: Topic :: Scientific/Engineering
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Requires-Dist: pydantic>=2.13.1
+Requires-Dist: pyyaml>=6.0.2 ; extra == 'cli'
 Requires-Dist: questionary>=2.1.1 ; extra == 'cli'
 Requires-Dist: typer>=0.16.0 ; extra == 'cli'
 Requires-Python: >=3.12
@@ -599,6 +600,91 @@ These commands prompt for missing details, write `*.annotation.json` or `data-an
 and optionally derive README sidecars. Post-hoc records are marked with
 `capture_mode="post_hoc"`.
 
+For shell workflows, you can move the prompt answers into a YAML file and run
+the command non-interactively:
+
+```bash
+data-annotations annotate file path/to/participants.csv --answers participants.yaml
+data-annotations annotate directory path/to/run-001 --answers run-001.yaml
+data-annotations annotate answers check participants.yaml
+```
+
+When `--answers` is provided, `--no-interactive` is the default. Use
+`--interactive` if you want the YAML file to provide defaults and still prompt
+for missing required values. If the YAML file includes `target`, the positional
+target may be omitted; when both are provided, they must resolve to the same
+path. Environment variables such as `$DATA_ROOT` and `${DATA_ROOT}` are expanded
+inside string values, and validation fails if a referenced variable is not set.
+The `answers check` helper requires `target` so it can infer whether the answers
+describe a file or a directory.
+
+File answers can use top-level prompt-style keys:
+
+```yaml
+target: path/to/participants.csv
+title: Participant Cohort
+summary: Participant-level cohort assignments.
+kind: dataset
+
+inputs:
+  - ${DATA_ROOT}/raw/participants.csv
+
+params:
+  split: validation
+
+provenance:
+  command: bash scripts/build_participants.sh
+  script: scripts/build_participants.sh
+  git_sha: deadbeef
+
+fields:
+  - name: participant_id
+    summary: Stable participant identifier.
+    data_type: string
+    required: true
+    nullable: false
+
+primary_key:
+  - participant_id
+```
+
+Directory answers use an explicit inventory. Paths in `artifacts`,
+`artifact_groups.paths`, and `child_bundles` are relative to the annotated
+directory unless absolute:
+
+```yaml
+target: path/to/run-001
+title: Processing outputs
+summary: Files produced by the shell processing workflow.
+
+provenance:
+  command: bash process_from_instrument.sh
+  script: process_from_instrument.sh
+
+artifacts:
+  - path: processed.csv
+    kind: dataset
+    title: Processed instrument output
+    summary: Normalized output from the processing script.
+
+artifact_groups:
+  - title: Diagnostic plots
+    kind: plot
+    selector: plots/*.png
+    paths:
+      - plots/qc-1.png
+      - plots/qc-2.png
+
+child_bundles:
+  - path: model
+    annotation_path: model/data-annotations.json
+```
+
+Answers files may also use schema-style aliases such as `subject.path`,
+`subject.kind`, `description.title`, `description.summary`,
+`description.artifacts`, `description.artifact_groups`, `provenance.inputs`,
+and `provenance.params`.
+
 When group selectors are provided, the CLI expands them to concrete member paths
 at annotation time. Grouped files are tracked in `subject.produced_files[]` but
 are skipped by the per-file prompt flow, so you do not have to answer the same
@@ -617,6 +703,10 @@ data-annotations provenance chain path/to/artifact --full-paths
 data-annotations provenance checkout path/to/artifact
 ```
 
+Command `checkout` downloads the recorded Git remote and checks out the recorded
+commit. It prompts before downloading source code and defaults to No; use
+`--force` when running trusted provenance checkout non-interactively.
+
 Command `match` auto-discovers `*.annotation.json` for files and `data-annotations.json` for
 directories, prints a verification summary, and suggests the exact `checkout`
 command to run next when Git recovery metadata is available.
@@ -626,6 +716,27 @@ those inputs. Its default output shows a compact relative-path tree and lists
 stale, missing, or unverifiable nodes first; use `--full-paths` when you need
 absolute paths.
 
+For publication workflows, create a sanitized copy of an annotated artifact tree:
+
+```bash
+data-annotations publish path/to/run-001 path/to/publish-bundle
+data-annotations publish path/to/run-001 path/to/publish-bundle \
+  --prefix /private/raw/study-a='$INPUT_ROOT'
+data-annotations publish path/to/run-001 path/to/publish-metadata \
+  --annotations-only
+data-annotations publish path/to/run-001 path/to/publish-bundle --dry-run
+```
+
+Command `publish` recursively discovers file annotations (`*.annotation.json`) and
+directory annotations (`data-annotations.json`), writes a mirrored publish bundle,
+and regenerates README sidecars from sanitized annotation JSON. Paths under the
+source directory are rewritten to `$ARTIFACT_ROOT/...`; additional `--prefix`
+mappings rewrite other private path roots. Hostname, username, and SLURM job ID
+are redacted by default. Git remote URLs are preserved unless
+`--redact-git-remote` is provided. Strict mode is enabled by default and fails if
+any local absolute path remains after sanitization; use `--no-strict` only after
+reviewing `--dry-run` output.
+
 If `data-annotations provenance --help` does not list `chain`, your shell is
 resolving an older installed command. From a source checkout, use
 `uv run data-annotations provenance chain ...`, or reinstall the CLI from the
@@ -656,6 +767,7 @@ uv run data-annotations annotate directory path/to/run-001
 uv run data-annotations provenance match path/to/participants.csv
 uv run data-annotations provenance chain path/to/participants.csv
 uv run data-annotations provenance checkout path/to/participants.csv
+uv run data-annotations publish path/to/run-001 path/to/publish-bundle
 ```
 
 ## API Overview
@@ -727,6 +839,13 @@ uv run data-annotations provenance checkout path/to/participants.csv
 - `artifact_matches_manifest(...)`
 - `checkout_manifest_source(...)`
 
+### Publish Functions
+
+- `discover_annotation_paths(...)`
+- `sanitize_annotation_document(...)`
+- `sanitize_annotation_path(...)`
+- `publish_directory(...)`
+
 ## Examples
 
 Runnable examples live in `examples/` and mirror the README workflows.
@@ -741,6 +860,7 @@ uv run python examples/record_file_description.py
 uv run python examples/record_directory_description.py
 uv run python examples/annotate_file.py
 uv run python examples/annotate_directory.py
+uv run python examples/annotate_file_answers_cli.py
 uv run python examples/write_file_manifest.py
 uv run python examples/write_directory_manifest.py
 uv run python examples/write_file_description.py
@@ -749,6 +869,7 @@ uv run python examples/provenance_chain.py
 uv run python examples/provenance_chain_cli.py
 uv run python examples/recover_provenance.py
 uv run python examples/recover_provenance_cli.py
+uv run python examples/publish_cli.py
 ```
 
 Each example writes its outputs to a fresh temporary directory and prints the

{data_annotations-2.3.0 → data_annotations-2.4.0}/README.md

@@ -570,6 +570,91 @@ These commands prompt for missing details, write `*.annotation.json` or `data-an
 and optionally derive README sidecars. Post-hoc records are marked with
 `capture_mode="post_hoc"`.
 
+For shell workflows, you can move the prompt answers into a YAML file and run
+the command non-interactively:
+
+```bash
+data-annotations annotate file path/to/participants.csv --answers participants.yaml
+data-annotations annotate directory path/to/run-001 --answers run-001.yaml
+data-annotations annotate answers check participants.yaml
+```
+
+When `--answers` is provided, `--no-interactive` is the default. Use
+`--interactive` if you want the YAML file to provide defaults and still prompt
+for missing required values. If the YAML file includes `target`, the positional
+target may be omitted; when both are provided, they must resolve to the same
+path. Environment variables such as `$DATA_ROOT` and `${DATA_ROOT}` are expanded
+inside string values, and validation fails if a referenced variable is not set.
+The `answers check` helper requires `target` so it can infer whether the answers
+describe a file or a directory.
+
+File answers can use top-level prompt-style keys:
+
+```yaml
+target: path/to/participants.csv
+title: Participant Cohort
+summary: Participant-level cohort assignments.
+kind: dataset
+
+inputs:
+  - ${DATA_ROOT}/raw/participants.csv
+
+params:
+  split: validation
+
+provenance:
+  command: bash scripts/build_participants.sh
+  script: scripts/build_participants.sh
+  git_sha: deadbeef
+
+fields:
+  - name: participant_id
+    summary: Stable participant identifier.
+    data_type: string
+    required: true
+    nullable: false
+
+primary_key:
+  - participant_id
+```
+
+Directory answers use an explicit inventory. Paths in `artifacts`,
+`artifact_groups.paths`, and `child_bundles` are relative to the annotated
+directory unless absolute:
+
+```yaml
+target: path/to/run-001
+title: Processing outputs
+summary: Files produced by the shell processing workflow.
+
+provenance:
+  command: bash process_from_instrument.sh
+  script: process_from_instrument.sh
+
+artifacts:
+  - path: processed.csv
+    kind: dataset
+    title: Processed instrument output
+    summary: Normalized output from the processing script.
+
+artifact_groups:
+  - title: Diagnostic plots
+    kind: plot
+    selector: plots/*.png
+    paths:
+      - plots/qc-1.png
+      - plots/qc-2.png
+
+child_bundles:
+  - path: model
+    annotation_path: model/data-annotations.json
+```
+
+Answers files may also use schema-style aliases such as `subject.path`,
+`subject.kind`, `description.title`, `description.summary`,
+`description.artifacts`, `description.artifact_groups`, `provenance.inputs`,
+and `provenance.params`.
+
 When group selectors are provided, the CLI expands them to concrete member paths
 at annotation time. Grouped files are tracked in `subject.produced_files[]` but
 are skipped by the per-file prompt flow, so you do not have to answer the same
@@ -588,6 +673,10 @@ data-annotations provenance chain path/to/artifact --full-paths
 data-annotations provenance checkout path/to/artifact
 ```
 
+Command `checkout` downloads the recorded Git remote and checks out the recorded
+commit. It prompts before downloading source code and defaults to No; use
+`--force` when running trusted provenance checkout non-interactively.
+
 Command `match` auto-discovers `*.annotation.json` for files and `data-annotations.json` for
 directories, prints a verification summary, and suggests the exact `checkout`
 command to run next when Git recovery metadata is available.
@@ -597,6 +686,27 @@ those inputs. Its default output shows a compact relative-path tree and lists
 stale, missing, or unverifiable nodes first; use `--full-paths` when you need
 absolute paths.
 
+For publication workflows, create a sanitized copy of an annotated artifact tree:
+
+```bash
+data-annotations publish path/to/run-001 path/to/publish-bundle
+data-annotations publish path/to/run-001 path/to/publish-bundle \
+  --prefix /private/raw/study-a='$INPUT_ROOT'
+data-annotations publish path/to/run-001 path/to/publish-metadata \
+  --annotations-only
+data-annotations publish path/to/run-001 path/to/publish-bundle --dry-run
+```
+
+Command `publish` recursively discovers file annotations (`*.annotation.json`) and
+directory annotations (`data-annotations.json`), writes a mirrored publish bundle,
+and regenerates README sidecars from sanitized annotation JSON. Paths under the
+source directory are rewritten to `$ARTIFACT_ROOT/...`; additional `--prefix`
+mappings rewrite other private path roots. Hostname, username, and SLURM job ID
+are redacted by default. Git remote URLs are preserved unless
+`--redact-git-remote` is provided. Strict mode is enabled by default and fails if
+any local absolute path remains after sanitization; use `--no-strict` only after
+reviewing `--dry-run` output.
+
 If `data-annotations provenance --help` does not list `chain`, your shell is
 resolving an older installed command. From a source checkout, use
 `uv run data-annotations provenance chain ...`, or reinstall the CLI from the
@@ -627,6 +737,7 @@ uv run data-annotations annotate directory path/to/run-001
 uv run data-annotations provenance match path/to/participants.csv
 uv run data-annotations provenance chain path/to/participants.csv
 uv run data-annotations provenance checkout path/to/participants.csv
+uv run data-annotations publish path/to/run-001 path/to/publish-bundle
 ```
 
 ## API Overview
@@ -698,6 +809,13 @@ uv run data-annotations provenance checkout path/to/participants.csv
 - `artifact_matches_manifest(...)`
 - `checkout_manifest_source(...)`
 
+### Publish Functions
+
+- `discover_annotation_paths(...)`
+- `sanitize_annotation_document(...)`
+- `sanitize_annotation_path(...)`
+- `publish_directory(...)`
+
 ## Examples
 
 Runnable examples live in `examples/` and mirror the README workflows.
@@ -712,6 +830,7 @@ uv run python examples/record_file_description.py
 uv run python examples/record_directory_description.py
 uv run python examples/annotate_file.py
 uv run python examples/annotate_directory.py
+uv run python examples/annotate_file_answers_cli.py
 uv run python examples/write_file_manifest.py
 uv run python examples/write_directory_manifest.py
 uv run python examples/write_file_description.py
@@ -720,6 +839,7 @@ uv run python examples/provenance_chain.py
 uv run python examples/provenance_chain_cli.py
 uv run python examples/recover_provenance.py
 uv run python examples/recover_provenance_cli.py
+uv run python examples/publish_cli.py
 ```
 
 Each example writes its outputs to a fresh temporary directory and prints the

{data_annotations-2.3.0 → data_annotations-2.4.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "data-annotations"
-version = "2.3.0"
+version = "2.4.0"
 description = "Annotate generated data artifacts"
 readme = "README.md"
 authors = [
@@ -30,7 +30,7 @@ Changelog = "https://gitlab.com/ceda-unibas/tools/data-annotations/-/blob/main/C
 Issues = "https://gitlab.com/ceda-unibas/tools/data-annotations/-/issues"
 
 [project.optional-dependencies]
-cli = ["questionary>=2.1.1", "typer>=0.16.0"]
+cli = ["PyYAML>=6.0.2", "questionary>=2.1.1", "typer>=0.16.0"]
 
 [project.scripts]
 data-annotations = "data_annotations.cli:main"

{data_annotations-2.3.0 → data_annotations-2.4.0}/src/data_annotations/cli.py

@@ -1,7 +1,7 @@
 from typing import Any
 
 _CLI_IMPORT_ERROR: ModuleNotFoundError | None = None
-_CLI_OPTIONAL_DEPENDENCIES = {"questionary", "typer"}
+_CLI_OPTIONAL_DEPENDENCIES = {"questionary", "typer", "yaml"}
 app: Any = None
 
 try:
@@ -9,6 +9,7 @@ try:
 
     from data_annotations.cli_app.annotate import annotate_app
     from data_annotations.cli_app.provenance_commands import provenance_app
+    from data_annotations.cli_app.publish import publish_command
 except ModuleNotFoundError as exc:
     if exc.name not in _CLI_OPTIONAL_DEPENDENCIES:
         raise
@@ -17,6 +18,7 @@ else:
     app = typer.Typer(no_args_is_help=True)
     app.add_typer(annotate_app, name="annotate")
     app.add_typer(provenance_app, name="provenance")
+    app.command("publish")(publish_command)
 
 
 def main() -> None: