data-annotations 2.4.0__tar.gz → 2.5.0__tar.gz

{data_annotations-2.4.0 → data_annotations-2.5.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: data-annotations
-Version: 2.4.0
-Summary: Annotate generated data artifacts
+Version: 2.5.0
+Summary: Annotate data artifacts with provenance and descriptions
 Keywords: annotations,data,metadata,provenance,reproducibility
 Author: Rodrigo C.  G.  Pena
 Author-email: Rodrigo C.  G.  Pena <rodrigo.cerqueiragonzalezpena@unibas.ch>
@@ -102,6 +102,8 @@ Every annotation document includes provenance with:
 - The script path relative to the Git repo root when it can be determined
 - Git commit, branch, dirty state, canonical repository remote, exact tags, and
   `git describe` output when available
+- A source-code reference for recovery, derived from Git metadata when possible
+  or supplied explicitly for archives, individual files, and DOI/URI records
 - The current `SLURM_JOB_ID` when available
 - Structured snapshots for recorded local inputs, including file checksums,
   directory content digests, and upstream annotation sidecar references when
@@ -110,8 +112,8 @@ Every annotation document includes provenance with:
 You can also attach your own parameters, input file paths, and function names.
 Local filesystem paths in provenance are stored as absolute paths. URI-style inputs
 such as `s3://...` or `https://...` are preserved as provided.
-Git tags and `git_describe` are human-friendly hints only; `git_sha` remains the
-source of truth for reproducibility, matching, and source checkout.
+Git tags and `git_describe` are human-friendly hints only. For Git sources,
+`git_sha` and `source_code.revision` identify the recoverable code state.
 
 ## Quick Start
 
@@ -537,8 +539,9 @@ per call.
 Use `artifact_matches_manifest(...)` to verify whether a detached artifact still
 matches an annotation document. Use `analyze_provenance_chain(...)` when you also
 want to verify recorded inputs and recursively follow upstream annotation
-sidecars. Use `checkout_manifest_source(...)` to recover the recorded code state
-from Git metadata.
+sidecars. Use `recover_manifest_source(...)` to recover the recorded source code
+from Git metadata, a recorded source archive, or a recorded source file.
+`checkout_manifest_source(...)` remains available as a compatibility alias.
 
 ```python
 from pathlib import Path
@@ -546,7 +549,7 @@ from pathlib import Path
 from data_annotations.provenance import (
     analyze_provenance_chain,
     artifact_matches_manifest,
-    checkout_manifest_source,
+    recover_manifest_source,
 )
 
 annotation_path = Path("outputs/participants.csv.annotation.json")
@@ -555,7 +558,7 @@ artifact_path = Path("downloads/participants.csv")
 if artifact_matches_manifest(artifact_path, annotation_path):
     chain = analyze_provenance_chain(artifact_path)
     print(chain.status)
-    recovered = checkout_manifest_source(annotation_path)
+    recovered = recover_manifest_source(annotation_path)
     print(recovered.checkout_path)
     print(recovered.script_path)
 ```
@@ -572,9 +575,9 @@ still attach provenance and description after the fact.
 
 Post-hoc descriptions can still be very useful, but the quality of post-hoc
 provenance depends on how exact the supplied answers are. In particular, fields
-such as the generating script, command, function, Git commit, repository path,
-Git tags, `git describe` output, inputs, and parameters are only as reliable as
-the information entered during annotation.
+such as the generating script, command, function, source-code URI, Git commit,
+repository path, Git tags, `git describe` output, inputs, and parameters are
+only as reliable as the information entered during annotation.
 
 ## CLI Workflow
 
@@ -627,25 +630,31 @@ summary: Participant-level cohort assignments.
 kind: dataset
 
 inputs:
-  - ${DATA_ROOT}/raw/participants.csv
+    - ${DATA_ROOT}/raw/participants.csv
 
 params:
-  split: validation
+    split: validation
 
 provenance:
-  command: bash scripts/build_participants.sh
-  script: scripts/build_participants.sh
-  git_sha: deadbeef
+    command: bash scripts/build_participants.sh
+    script: scripts/build_participants.sh
+    git_sha: deadbeef
+    source_code:
+        kind: archive
+        uri: https://doi.org/10.5281/zenodo.12345
+        download_uri: https://zenodo.org/records/12345/files/source.zip
+        path: scripts/build_participants.sh
+        sha256: 0000000000000000000000000000000000000000000000000000000000000000
 
 fields:
-  - name: participant_id
-    summary: Stable participant identifier.
-    data_type: string
-    required: true
-    nullable: false
+    - name: participant_id
+      summary: Stable participant identifier.
+      data_type: string
+      required: true
+      nullable: false
 
 primary_key:
-  - participant_id
+    - participant_id
 ```
 
 Directory answers use an explicit inventory. Paths in `artifacts`,
@@ -658,26 +667,26 @@ title: Processing outputs
 summary: Files produced by the shell processing workflow.
 
 provenance:
-  command: bash process_from_instrument.sh
-  script: process_from_instrument.sh
+    command: bash process_from_instrument.sh
+    script: process_from_instrument.sh
 
 artifacts:
-  - path: processed.csv
-    kind: dataset
-    title: Processed instrument output
-    summary: Normalized output from the processing script.
+    - path: processed.csv
+      kind: dataset
+      title: Processed instrument output
+      summary: Normalized output from the processing script.
 
 artifact_groups:
-  - title: Diagnostic plots
-    kind: plot
-    selector: plots/*.png
-    paths:
-      - plots/qc-1.png
-      - plots/qc-2.png
+    - title: Diagnostic plots
+      kind: plot
+      selector: plots/*.png
+      paths:
+          - plots/qc-1.png
+          - plots/qc-2.png
 
 child_bundles:
-  - path: model
-    annotation_path: model/data-annotations.json
+    - path: model
+      annotation_path: model/data-annotations.json
 ```
 
 Answers files may also use schema-style aliases such as `subject.path`,
@@ -685,14 +694,24 @@ Answers files may also use schema-style aliases such as `subject.path`,
 `description.artifacts`, `description.artifact_groups`, `provenance.inputs`,
 and `provenance.params`.
 
+For source-code recovery, `provenance.source_code.kind` may be `git`, `archive`,
+`file`, or `uri`. Git sources use `uri` plus `revision`; archive and file
+sources use `uri` or `download_uri` plus an optional `sha256`; `path` points to
+the generating script inside the recovered source. DOI or landing-page-only
+references can be recorded with `kind: uri`, but they are not directly
+recoverable unless a direct archive or file `download_uri` is also recorded.
+
 When group selectors are provided, the CLI expands them to concrete member paths
 at annotation time. Grouped files are tracked in `subject.produced_files[]` but
 are skipped by the per-file prompt flow, so you do not have to answer the same
 questions for every matching file.
 
-For post-hoc provenance, use repeatable `--git-tag` and optional
-`--git-describe` when you know the original code state. These values are stored
-as human-readable hints; `--git-sha` remains the field used for recovery.
+For post-hoc provenance, use `--source-kind`, `--source-uri`,
+`--source-download-uri`, `--source-path`, `--source-revision`, and
+`--source-sha256` when the generating code is recoverable from a Git remote,
+source archive, source file, or reference URI. Use repeatable `--git-tag` and
+optional `--git-describe` when you know the original Git state; these values are
+stored as human-readable hints.
 
 For provenance inspection and source recovery:
 
@@ -703,8 +722,12 @@ data-annotations provenance chain path/to/artifact --full-paths
 data-annotations provenance checkout path/to/artifact
 ```
 
-Command `checkout` downloads the recorded Git remote and checks out the recorded
-commit. It prompts before downloading source code and defaults to No; use
+Command `checkout` recovers the recorded source code. For Git sources, it clones
+the recorded remote and checks out the recorded revision. For archive and file
+sources, it downloads or copies the recorded object, verifies `sha256` when
+present, and resolves the generating script path when recorded. Reference-only
+URI sources are preserved in the annotation but are not directly recoverable.
+The command prompts before downloading source code and defaults to No; use
 `--force` when running trusted provenance checkout non-interactively.
 
 Command `match` auto-discovers `*.annotation.json` for files and `data-annotations.json` for
@@ -820,6 +843,8 @@ uv run data-annotations publish path/to/run-001 path/to/publish-bundle
 - `ProducedFile`
 - `ChildBundle`
 - `InputArtifact`
+- `SourceCodeKind`
+- `SourceCodeReference`
 - `BaseProvenance`
 - `FileManifest`
 - `DirectoryManifest`
@@ -837,6 +862,7 @@ uv run data-annotations publish path/to/run-001 path/to/publish-bundle
 - `analyze_provenance_chain(...)`
 - `provenance_chain_is_fresh(...)`
 - `artifact_matches_manifest(...)`
+- `recover_manifest_source(...)`
 - `checkout_manifest_source(...)`
 
 ### Publish Functions
@@ -869,6 +895,7 @@ uv run python examples/provenance_chain.py
 uv run python examples/provenance_chain_cli.py
 uv run python examples/recover_provenance.py
 uv run python examples/recover_provenance_cli.py
+uv run python examples/recover_archive_source.py
 uv run python examples/publish_cli.py
 ```
 

{data_annotations-2.4.0 → data_annotations-2.5.0}/README.md

@@ -72,6 +72,8 @@ Every annotation document includes provenance with:
 - The script path relative to the Git repo root when it can be determined
 - Git commit, branch, dirty state, canonical repository remote, exact tags, and
   `git describe` output when available
+- A source-code reference for recovery, derived from Git metadata when possible
+  or supplied explicitly for archives, individual files, and DOI/URI records
 - The current `SLURM_JOB_ID` when available
 - Structured snapshots for recorded local inputs, including file checksums,
   directory content digests, and upstream annotation sidecar references when
@@ -80,8 +82,8 @@ Every annotation document includes provenance with:
 You can also attach your own parameters, input file paths, and function names.
 Local filesystem paths in provenance are stored as absolute paths. URI-style inputs
 such as `s3://...` or `https://...` are preserved as provided.
-Git tags and `git_describe` are human-friendly hints only; `git_sha` remains the
-source of truth for reproducibility, matching, and source checkout.
+Git tags and `git_describe` are human-friendly hints only. For Git sources,
+`git_sha` and `source_code.revision` identify the recoverable code state.
 
 ## Quick Start
 
@@ -507,8 +509,9 @@ per call.
 Use `artifact_matches_manifest(...)` to verify whether a detached artifact still
 matches an annotation document. Use `analyze_provenance_chain(...)` when you also
 want to verify recorded inputs and recursively follow upstream annotation
-sidecars. Use `checkout_manifest_source(...)` to recover the recorded code state
-from Git metadata.
+sidecars. Use `recover_manifest_source(...)` to recover the recorded source code
+from Git metadata, a recorded source archive, or a recorded source file.
+`checkout_manifest_source(...)` remains available as a compatibility alias.
 
 ```python
 from pathlib import Path
@@ -516,7 +519,7 @@ from pathlib import Path
 from data_annotations.provenance import (
     analyze_provenance_chain,
     artifact_matches_manifest,
-    checkout_manifest_source,
+    recover_manifest_source,
 )
 
 annotation_path = Path("outputs/participants.csv.annotation.json")
@@ -525,7 +528,7 @@ artifact_path = Path("downloads/participants.csv")
 if artifact_matches_manifest(artifact_path, annotation_path):
     chain = analyze_provenance_chain(artifact_path)
     print(chain.status)
-    recovered = checkout_manifest_source(annotation_path)
+    recovered = recover_manifest_source(annotation_path)
     print(recovered.checkout_path)
     print(recovered.script_path)
 ```
@@ -542,9 +545,9 @@ still attach provenance and description after the fact.
 
 Post-hoc descriptions can still be very useful, but the quality of post-hoc
 provenance depends on how exact the supplied answers are. In particular, fields
-such as the generating script, command, function, Git commit, repository path,
-Git tags, `git describe` output, inputs, and parameters are only as reliable as
-the information entered during annotation.
+such as the generating script, command, function, source-code URI, Git commit,
+repository path, Git tags, `git describe` output, inputs, and parameters are
+only as reliable as the information entered during annotation.
 
 ## CLI Workflow
 
@@ -597,25 +600,31 @@ summary: Participant-level cohort assignments.
 kind: dataset
 
 inputs:
-  - ${DATA_ROOT}/raw/participants.csv
+    - ${DATA_ROOT}/raw/participants.csv
 
 params:
-  split: validation
+    split: validation
 
 provenance:
-  command: bash scripts/build_participants.sh
-  script: scripts/build_participants.sh
-  git_sha: deadbeef
+    command: bash scripts/build_participants.sh
+    script: scripts/build_participants.sh
+    git_sha: deadbeef
+    source_code:
+        kind: archive
+        uri: https://doi.org/10.5281/zenodo.12345
+        download_uri: https://zenodo.org/records/12345/files/source.zip
+        path: scripts/build_participants.sh
+        sha256: 0000000000000000000000000000000000000000000000000000000000000000
 
 fields:
-  - name: participant_id
-    summary: Stable participant identifier.
-    data_type: string
-    required: true
-    nullable: false
+    - name: participant_id
+      summary: Stable participant identifier.
+      data_type: string
+      required: true
+      nullable: false
 
 primary_key:
-  - participant_id
+    - participant_id
 ```
 
 Directory answers use an explicit inventory. Paths in `artifacts`,
@@ -628,26 +637,26 @@ title: Processing outputs
 summary: Files produced by the shell processing workflow.
 
 provenance:
-  command: bash process_from_instrument.sh
-  script: process_from_instrument.sh
+    command: bash process_from_instrument.sh
+    script: process_from_instrument.sh
 
 artifacts:
-  - path: processed.csv
-    kind: dataset
-    title: Processed instrument output
-    summary: Normalized output from the processing script.
+    - path: processed.csv
+      kind: dataset
+      title: Processed instrument output
+      summary: Normalized output from the processing script.
 
 artifact_groups:
-  - title: Diagnostic plots
-    kind: plot
-    selector: plots/*.png
-    paths:
-      - plots/qc-1.png
-      - plots/qc-2.png
+    - title: Diagnostic plots
+      kind: plot
+      selector: plots/*.png
+      paths:
+          - plots/qc-1.png
+          - plots/qc-2.png
 
 child_bundles:
-  - path: model
-    annotation_path: model/data-annotations.json
+    - path: model
+      annotation_path: model/data-annotations.json
 ```
 
 Answers files may also use schema-style aliases such as `subject.path`,
@@ -655,14 +664,24 @@ Answers files may also use schema-style aliases such as `subject.path`,
 `description.artifacts`, `description.artifact_groups`, `provenance.inputs`,
 and `provenance.params`.
 
+For source-code recovery, `provenance.source_code.kind` may be `git`, `archive`,
+`file`, or `uri`. Git sources use `uri` plus `revision`; archive and file
+sources use `uri` or `download_uri` plus an optional `sha256`; `path` points to
+the generating script inside the recovered source. DOI or landing-page-only
+references can be recorded with `kind: uri`, but they are not directly
+recoverable unless a direct archive or file `download_uri` is also recorded.
+
 When group selectors are provided, the CLI expands them to concrete member paths
 at annotation time. Grouped files are tracked in `subject.produced_files[]` but
 are skipped by the per-file prompt flow, so you do not have to answer the same
 questions for every matching file.
 
-For post-hoc provenance, use repeatable `--git-tag` and optional
-`--git-describe` when you know the original code state. These values are stored
-as human-readable hints; `--git-sha` remains the field used for recovery.
+For post-hoc provenance, use `--source-kind`, `--source-uri`,
+`--source-download-uri`, `--source-path`, `--source-revision`, and
+`--source-sha256` when the generating code is recoverable from a Git remote,
+source archive, source file, or reference URI. Use repeatable `--git-tag` and
+optional `--git-describe` when you know the original Git state; these values are
+stored as human-readable hints.
 
 For provenance inspection and source recovery:
 
@@ -673,8 +692,12 @@ data-annotations provenance chain path/to/artifact --full-paths
 data-annotations provenance checkout path/to/artifact
 ```
 
-Command `checkout` downloads the recorded Git remote and checks out the recorded
-commit. It prompts before downloading source code and defaults to No; use
+Command `checkout` recovers the recorded source code. For Git sources, it clones
+the recorded remote and checks out the recorded revision. For archive and file
+sources, it downloads or copies the recorded object, verifies `sha256` when
+present, and resolves the generating script path when recorded. Reference-only
+URI sources are preserved in the annotation but are not directly recoverable.
+The command prompts before downloading source code and defaults to No; use
 `--force` when running trusted provenance checkout non-interactively.
 
 Command `match` auto-discovers `*.annotation.json` for files and `data-annotations.json` for
@@ -790,6 +813,8 @@ uv run data-annotations publish path/to/run-001 path/to/publish-bundle
 - `ProducedFile`
 - `ChildBundle`
 - `InputArtifact`
+- `SourceCodeKind`
+- `SourceCodeReference`
 - `BaseProvenance`
 - `FileManifest`
 - `DirectoryManifest`
@@ -807,6 +832,7 @@ uv run data-annotations publish path/to/run-001 path/to/publish-bundle
 - `analyze_provenance_chain(...)`
 - `provenance_chain_is_fresh(...)`
 - `artifact_matches_manifest(...)`
+- `recover_manifest_source(...)`
 - `checkout_manifest_source(...)`
 
 ### Publish Functions
@@ -839,6 +865,7 @@ uv run python examples/provenance_chain.py
 uv run python examples/provenance_chain_cli.py
 uv run python examples/recover_provenance.py
 uv run python examples/recover_provenance_cli.py
+uv run python examples/recover_archive_source.py
 uv run python examples/publish_cli.py
 ```
 

{data_annotations-2.4.0 → data_annotations-2.5.0}/pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "data-annotations"
-version = "2.4.0"
-description = "Annotate generated data artifacts"
+version = "2.5.0"
+description = "Annotate data artifacts with provenance and descriptions"
 readme = "README.md"
 authors = [
   { name = "Rodrigo C.  G.  Pena", email = "rodrigo.cerqueiragonzalezpena@unibas.ch" },

{data_annotations-2.4.0 → data_annotations-2.5.0}/src/data_annotations/annotations/models.py

@@ -22,14 +22,14 @@ class DirectoryArtifactSubject(BaseModel):
 
 
 class FileAnnotationDocument(BaseModel):
-    annotation_version: Literal["5"] = "5"
+    annotation_version: Literal["6"] = "6"
     subject: FileArtifactSubject
     provenance: BaseProvenance
     description: FileDescription
 
 
 class DirectoryAnnotationDocument(BaseModel):
-    annotation_version: Literal["5"] = "5"
+    annotation_version: Literal["6"] = "6"
     subject: DirectoryArtifactSubject
     provenance: BaseProvenance
     description: DirectoryDescription