data-annotations 2.1.2__tar.gz → 2.3.0__tar.gz

{data_annotations-2.1.2 → data_annotations-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: data-annotations
-Version: 2.1.2
+Version: 2.3.0
 Summary: Annotate generated data artifacts
 Keywords: annotations,data,metadata,provenance,reproducibility
 Author: Rodrigo C.  G.  Pena
@@ -29,7 +29,7 @@ Description-Content-Type: text/markdown
 
 # data-annotations
 
-A small Python package for attaching provenance and structured descriptions to the
+A Python package for attaching provenance and structured descriptions to the
 files and directories your workflows produce.
 
 It is designed for lightweight research and reproducibility pipelines where you want
@@ -37,11 +37,11 @@ generated datasets, tables, plots, or reports to carry enough context to explain
 where they came from and what they contain.
 
 The package captures common provenance automatically and writes plain JSON and
-Markdown artifacts that are easy to inspect or archive. The canonical on-disk format
-is now a single annotation document:
+Markdown artifacts that are easy to inspect or archive. The canonical on-disk
+format uses one JSON annotation document per artifact:
 
-- Files use `artifact.ext.meta.json`
-- Directories use `manifest.json`
+- Files use `artifact.ext.annotation.json`
+- Directories carry `data-annotations.json` at their root
 
 Each annotation document stores four top-level sections:
 
@@ -50,6 +50,10 @@ Each annotation document stores four top-level sections:
 - `provenance`
 - `description`
 
+Here's the mental model: files get a visible sibling annotation, and
+directories carry one visible annotation at their root. Treat the annotation as
+part of the research output bundle.
+
 See the [changelog](CHANGELOG.md) for release history and upgrade-oriented notes.
 
 ## Installation
@@ -95,12 +99,18 @@ Every annotation document includes provenance with:
 - Hostname and username
 - The script path and command-line arguments
 - The script path relative to the Git repo root when it can be determined
-- Git commit, branch, dirty state, and canonical repository remote when available
+- Git commit, branch, dirty state, canonical repository remote, exact tags, and
+  `git describe` output when available
 - The current `SLURM_JOB_ID` when available
+- Structured snapshots for recorded local inputs, including file checksums,
+  directory content digests, and upstream annotation sidecar references when
+  present
 
 You can also attach your own parameters, input file paths, and function names.
 Local filesystem paths in provenance are stored as absolute paths. URI-style inputs
 such as `s3://...` or `https://...` are preserved as provided.
+Git tags and `git_describe` are human-friendly hints only; `git_sha` remains the
+source of truth for reproducibility, matching, and source checkout.
 
 ## Quick Start
 
@@ -111,7 +121,7 @@ provenance and emit sidecars automatically.
 
 For example, here is a complete file-level annotation workflow using the
 `record_file_annotation(...)` decorator. Once `write_participants` is called, it
-automatically generates sidecars `participants.csv.meta.json` and `participants.csv.README.md`.
+automatically generates sidecars `participants.csv.annotation.json` and `participants.csv.README.md`.
 The JSON sidecar will contain provenance and description metadata, and the Markdown sidecar
 will have a human-friendly rendering of the description provided in the decorator.
 
@@ -182,7 +192,7 @@ write_participants(
     split="validation",
 )
 
-print(f"{artifact_path}.meta.json")
+print(f"{artifact_path}.annotation.json")
 print(f"{artifact_path}.README.md")
 ```
 
@@ -235,7 +245,12 @@ Accepted directory return items are:
 
 - `DocumentedArtifact` when you want per-artifact title, summary, fields,
   keys, or missing-value metadata.
+- `DocumentedArtifactGroup` for `record_directory_annotation(...)` and
+  `record_directory_description(...)` when many files share one title, summary,
+  kind, and optional schema metadata.
 - `ProducedFile` when you only need path, kind, and optional precomputed hash.
+- `ChildBundle` when an annotated child directory should be referenced as its
+  own independently shareable bundle.
 - `(path, kind)` tuples when path and artifact kind are enough.
 - plain path-like values when the artifact kind can default to `"other"`.
 
@@ -249,7 +264,11 @@ Here is another decorator pattern example with `record_directory_annotation(...)
 from pathlib import Path
 
 from data_annotations.annotations import record_directory_annotation
-from data_annotations.description import DocumentedArtifact, FieldDefinition
+from data_annotations.description import (
+    DocumentedArtifact,
+    DocumentedArtifactGroup,
+    FieldDefinition,
+)
 from data_annotations.provenance import ProducedFile
 
 @record_directory_annotation(
@@ -294,13 +313,16 @@ def build_outputs(
         encoding="utf-8",
     )
 
-    plot_path = output_dir / "roc.png"
-    plot_path.write_bytes(
-        (
-            f"plot placeholder derived from {input_path.name} "
-            f"({len(participant_ids)} participants)\n"
-        ).encode("utf-8")
-    )
+    plot_paths = []
+    for day in ["2024-01-01", "2024-01-02", "2024-01-03"]:
+        plot_path = output_dir / f"sma_{day}.png"
+        plot_path.write_bytes(
+            (
+                f"plot placeholder for the SMA variable on {day}, "
+                f"derived from {input_path.name}\n"
+            ).encode("utf-8")
+        )
+        plot_paths.append(plot_path)
 
     return [
         DocumentedArtifact(
@@ -321,7 +343,13 @@ def build_outputs(
             ],
         ),
         ProducedFile(path=str(report_path), kind="report"),
-        (plot_path, "plot"),
+        DocumentedArtifactGroup(
+            title="Daily SMA plots",
+            summary="Plots of the same variable on different days.",
+            kind="plot",
+            paths=[str(path) for path in plot_paths],
+            selector="sma_*.png",
+        ),
     ]
 
 
@@ -332,7 +360,7 @@ build_outputs(
     split="validation",
 )
 
-print(output_dir / "manifest.json")
+print(output_dir / "data-annotations.json")
 print(output_dir / "README.md")
 ```
 
@@ -355,6 +383,7 @@ File annotations store:
 - `subject.kind`
 - `subject.sha256`
 - `provenance.*`
+- `provenance.input_artifacts[]`
 - `description.title`
 - `description.summary`
 - `description.fields`
@@ -368,16 +397,67 @@ Directory annotations store:
 
 - `subject.path`
 - `subject.produced_files[]`
+- `subject.child_bundles[]`
+- `subject.content_digest`
 - `provenance.*`
+- `provenance.input_artifacts[]`
 - `description.title`
 - `description.summary`
+- `description.artifact_groups[]`
 - `description.artifacts[]`
 - `description.acquisition_context`
 - `description.generation_context`
 - `description.description_updated_at`
 
-The `description` section intentionally excludes provenance linkage fields and
-file kinds for directory artifacts. Kinds live in `subject.produced_files`.
+Use `description.artifact_groups[]` when many files have the same meaning, and
+use `description.artifacts[]` only for file-specific notes, overrides, or schema.
+Groups are descriptive only. Integrity still lives in `subject.produced_files[]`,
+which tracks every concrete file by path, kind, and checksum.
+
+The `description` section intentionally excludes provenance linkage fields.
+Directory `produced_files[].path` values are stored relative to `subject.path`,
+which keeps verification stable when a complete output directory is copied or
+archived elsewhere. `subject.content_digest` is computed from sorted tracked file
+paths, file checksums, and referenced child bundle digests.
+
+## Artifact Groups
+
+Artifact groups are for homogeneous sets of files that researchers naturally
+understand as one output family: for example, 100 PNG plots of the same variable,
+one per acquisition day. A group stores the shared title, summary, kind, optional
+schema fields, and the concrete member paths. It can also store an informational
+`selector`, such as `plots/*.png`, to show how the group was chosen.
+
+Rules of thumb:
+
+- Use artifact groups when many files have the same meaning.
+- Use individual artifacts for file-specific notes, exceptions, or overrides.
+- It is OK for an individual artifact to also appear in a group.
+- Do not rely on groups for integrity. `subject.produced_files[]` remains the
+  complete checksum inventory.
+
+## Nested Directory Policy
+
+Annotate the smallest thing you would share as a unit. If a directory is one
+research output, give that directory one `data-annotations.json`, even when its
+tracked files live in nested subdirectories.
+
+Use recursive directory annotations for one bundle with nested files:
+
+```bash
+data-annotations annotate directory path/to/run-001 --recursive
+data-annotations annotate directory path/to/run-001 --max-depth 2
+```
+
+Use child bundle annotations when a subdirectory is independently meaningful,
+shareable, or reusable. In that case, annotate the child directory first, then
+annotate the parent. The parent records a compact `child_bundles[]` reference
+with the child path, child annotation path, and child content digest; it does not
+copy the child file inventory into the parent JSON.
+
+Post-hoc directory discovery follows the same rule. `--recursive` discovers
+nested files, but it stops at annotated child directories containing
+`data-annotations.json` and records them as child bundles.
 
 ## Provenance Decorators And Writers
 
@@ -412,7 +492,9 @@ write_report(
 
 Use `record_directory_manifest(...)` for directory outputs. Directory decorators
 accept `DocumentedArtifact`, `ProducedFile`, `(path, kind)`, and plain path-like
-return values.
+return values. Provenance-only APIs do not accept description groups; use
+unified annotation or description APIs when groups should appear in the JSON or
+README.
 
 If you want the direct writer approach instead, use `write_file_manifest(...)` and
 `write_directory_manifest(...)` (see `examples/`).
@@ -428,7 +510,9 @@ Key public description models:
 - `AllowedValue`
 - `FieldDefinition`
 - `DocumentedArtifact`
+- `DocumentedArtifactGroup`
 - `ArtifactDescription`
+- `ArtifactGroupDescription`
 - `FileDescription`
 - `DirectoryDescription`
 
@@ -450,21 +534,26 @@ per call.
 ## Recovery Helpers
 
 Use `artifact_matches_manifest(...)` to verify whether a detached artifact still
-matches an annotation document, and `checkout_manifest_source(...)` to recover the
-recorded code state from Git metadata.
+matches an annotation document. Use `analyze_provenance_chain(...)` when you also
+want to verify recorded inputs and recursively follow upstream annotation
+sidecars. Use `checkout_manifest_source(...)` to recover the recorded code state
+from Git metadata.
 
 ```python
 from pathlib import Path
 
 from data_annotations.provenance import (
+    analyze_provenance_chain,
     artifact_matches_manifest,
     checkout_manifest_source,
 )
 
-annotation_path = Path("outputs/participants.csv.meta.json")
+annotation_path = Path("outputs/participants.csv.annotation.json")
 artifact_path = Path("downloads/participants.csv")
 
 if artifact_matches_manifest(artifact_path, annotation_path):
+    chain = analyze_provenance_chain(artifact_path)
+    print(chain.status)
     recovered = checkout_manifest_source(annotation_path)
     print(recovered.checkout_path)
     print(recovered.script_path)
@@ -483,8 +572,8 @@ still attach provenance and description after the fact.
 Post-hoc descriptions can still be very useful, but the quality of post-hoc
 provenance depends on how exact the supplied answers are. In particular, fields
 such as the generating script, command, function, Git commit, repository path,
-inputs, and parameters are only as reliable as the information entered during
-annotation.
+Git tags, `git describe` output, inputs, and parameters are only as reliable as
+the information entered during annotation.
 
 ## CLI Workflow
 
@@ -496,22 +585,51 @@ For post-hoc annotation:
 ```bash
 data-annotations annotate file path/to/participants.csv
 data-annotations annotate directory path/to/run-001
+data-annotations annotate directory path/to/run-001 --recursive
+data-annotations annotate directory path/to/run-001 --max-depth 2
+data-annotations annotate directory path/to/run-001 \
+  --recursive \
+  --group-selector "plots/*.png" \
+  --group-title "Daily SMA plots" \
+  --group-summary "Plots of the same variable on different days." \
+  --group-kind plot
 ```
 
-These commands prompt for missing details, write `*.meta.json` or `manifest.json`,
+These commands prompt for missing details, write `*.annotation.json` or `data-annotations.json`,
 and optionally derive README sidecars. Post-hoc records are marked with
 `capture_mode="post_hoc"`.
 
+When group selectors are provided, the CLI expands them to concrete member paths
+at annotation time. Grouped files are tracked in `subject.produced_files[]` but
+are skipped by the per-file prompt flow, so you do not have to answer the same
+questions for every matching file.
+
+For post-hoc provenance, use repeatable `--git-tag` and optional
+`--git-describe` when you know the original code state. These values are stored
+as human-readable hints; `--git-sha` remains the field used for recovery.
+
 For provenance inspection and source recovery:
 
 ```bash
 data-annotations provenance match path/to/artifact
+data-annotations provenance chain path/to/artifact
+data-annotations provenance chain path/to/artifact --full-paths
 data-annotations provenance checkout path/to/artifact
 ```
 
-Command `match` auto-discovers `*.meta.json` for files and `manifest.json` for
+Command `match` auto-discovers `*.annotation.json` for files and `data-annotations.json` for
 directories, prints a verification summary, and suggests the exact `checkout`
 command to run next when Git recovery metadata is available.
+Command `chain` uses the same sidecar discovery, then verifies the artifact,
+recorded input snapshots, and any upstream annotation sidecars reachable from
+those inputs. Its default output shows a compact relative-path tree and lists
+stale, missing, or unverifiable nodes first; use `--full-paths` when you need
+absolute paths.
+
+If `data-annotations provenance --help` does not list `chain`, your shell is
+resolving an older installed command. From a source checkout, use
+`uv run data-annotations provenance chain ...`, or reinstall the CLI from the
+updated source before using the bare `data-annotations` command.
 
 ### Run With `uvx`
 
@@ -536,6 +654,7 @@ the project environment. You can then run:
 uv run data-annotations annotate file path/to/participants.csv
 uv run data-annotations annotate directory path/to/run-001
 uv run data-annotations provenance match path/to/participants.csv
+uv run data-annotations provenance chain path/to/participants.csv
 uv run data-annotations provenance checkout path/to/participants.csv
 ```
 
@@ -562,6 +681,17 @@ uv run data-annotations provenance checkout path/to/participants.csv
 - `annotate_file(...)`
 - `annotate_directory(...)`
 
+### Description Models
+
+- `AllowedValue`
+- `FieldDefinition`
+- `DocumentedArtifact`
+- `DocumentedArtifactGroup`
+- `ArtifactDescription`
+- `ArtifactGroupDescription`
+- `FileDescription`
+- `DirectoryDescription`
+
 ### Description Functions
 
 - `record_file_description(...)`
@@ -576,9 +706,13 @@ uv run data-annotations provenance checkout path/to/participants.csv
 ### Provenance Models
 
 - `ProducedFile`
+- `ChildBundle`
+- `InputArtifact`
 - `BaseProvenance`
 - `FileManifest`
 - `DirectoryManifest`
+- `ProvenanceChainNode`
+- `ProvenanceChainReport`
 - `RecoveredSource`
 
 ### Provenance Functions
@@ -587,6 +721,9 @@ uv run data-annotations provenance checkout path/to/participants.csv
 - `record_directory_manifest(...)`
 - `write_file_manifest(...)`
 - `write_directory_manifest(...)`
+- `directory_content_digest(...)`
+- `analyze_provenance_chain(...)`
+- `provenance_chain_is_fresh(...)`
 - `artifact_matches_manifest(...)`
 - `checkout_manifest_source(...)`
 
@@ -608,6 +745,8 @@ uv run python examples/write_file_manifest.py
 uv run python examples/write_directory_manifest.py
 uv run python examples/write_file_description.py
 uv run python examples/write_directory_description.py
+uv run python examples/provenance_chain.py
+uv run python examples/provenance_chain_cli.py
 uv run python examples/recover_provenance.py
 uv run python examples/recover_provenance_cli.py
 ```