dkist-processing-visp 5.7.11__tar.gz → 5.7.13__tar.gz

dkist_processing_visp-5.7.13/.rovodev/.review-agent.md

@@ -0,0 +1,537 @@
+# Custom Code Review Instructions for dkist-processing-* Repositories
+
+This document configures automated PR review guidance for DKIST science data processing repositories.
+It is intended for repos in the `dkist-processing-*` family, including:
+
+- `dkist-processing-core`: shared workflow/task abstractions used by other processing packages.
+- `dkist-processing-common`: common processing utilities and task bases used by instrument packages.
+- Instrument processing packages such as `dkist-processing-visp`, where most science pipeline changes occur.
+
+Focus comments on objective, code-reviewable findings. Avoid blocking a PR on subjective preferences unless the code clearly violates one of the conventions below or introduces a maintainability/bug risk.
+
+---
+
+## Review Priorities
+
+When reviewing a PR, prioritize:
+
+1. Correctness and data integrity bugs.
+2. Violations of repository architecture or dependency direction.
+3. Patterns that make future instrument-specific changes harder.
+4. Missing tests for changed behavior.
+5. Maintainability issues that are concrete and actionable.
+
+Do not repeat findings that are already clearly handled by linters, formatters, or type checkers unless the issue has architectural or runtime significance.
+
+---
+
+## Repository Layering and Dependency Direction
+
+**Description**: Preserve the dependency direction across processing repositories.
+
+- `dkist-processing-core` must remain generic and must not import from `dkist-processing-common` or any instrument package.
+- `dkist-processing-common` may depend on core abstractions, but must not import instrument-specific packages such as `dkist-processing-visp`.
+- Instrument packages may depend on common and core, but should not create cross-instrument dependencies.
+
+**Flag when**:
+
+- A lower-level package imports a higher-level package.
+- Shared code gains instrument-specific logic that belongs in an instrument package.
+- Instrument-specific constants, FITS keys, task names, or parser behavior are added to common/core without a clear abstraction.
+
+**Good pattern**:
+
+```python
+# instrument package code may use common abstractions
+from dkist_processing_common.tasks import WorkflowTaskBase
+```
+
+**Bad pattern**:
+
+```python
+# common/core code should not depend on an instrument package
+from dkist_processing_visp.models.tags import VispTag
+```
+
+---
+
+## Task Mixins
+
+**Description**: Task mixins encapsulate reusable task functionality in small modules and are composed into concrete task classes only where needed. Prefer minimizing new mixins and concentrating them around dependencies on external/shared resources rather than using mixins as general helper-method buckets.
+
+**Flag when**:
+
+- A mixin class does not end with `Mixin`.
+- A public mixin method is not prefixed with the mixin-specific namespace.
+  - Example: methods on `MetadataStoreMixin` should begin with `metadata_store_`.
+  - Example: methods on `ObjectStoreMixin` should begin with `object_store_`.
+- A mixin imports another mixin, creating hidden coupling between reusable behaviors.
+- A mixin contains behavior that is only meaningful for a single concrete task or instrument and would be clearer on that task/base class.
+
+**Good example**:
+
+```python
+class MetadataStoreMixin:
+    def metadata_store_get_parameter(self, key: str) -> object:
+        ...
+```
+
+**Bad example**:
+
+```python
+class MetadataStoreMixin:
+    def get_parameter(self, key: str) -> object:
+        ...
+```
+
+**Bad example**:
+
+```python
+# In a mixin module
+from dkist_processing_common.tasks.mixin.object_store import ObjectStoreMixin
+```
+
+---
+
+## Task Classes and Task Imports
+
+**Description**: Concrete task classes represent workflow/DAG nodes. Keep task dependencies explicit and avoid coupling concrete tasks to each other.
+
+**Flag when**:
+
+- A concrete task imports another concrete task from a `tasks` package module.
+- Imports from `tasks` package modules are not from `*_base` modules or the `tasks.mixin` package.
+- A class that is intended as a base class does not end with `Base`.
+- A concrete task contains branching logic that appears to select instrument-specific behavior that should instead be represented by separate task classes or instrument-specific subclasses.
+
+**Good example**:
+
+```python
+from dkist_processing_common.tasks.write_l1_base import WriteL1Base
+from dkist_processing_visp.tasks.mixin.beam_access import BeamAccessMixin
+```
+
+**Bad example**:
+
+```python
+from dkist_processing_visp.tasks.dark import DarkCalibration
+```
+
+---
+
+## Overriding Base-Class Methods
+
+**Description**: When overriding methods from task or framework base classes, preserve base-class behavior unless there is an explicit reason not to.
+
+**Flag when**:
+
+- A subclass overrides a method that exists on a base class and does not call `super().method_name(...)`.
+- The PR does not explain why the base implementation should be skipped.
+
+**Good example**:
+
+```python
+class ScienceCalibration(VispTaskBase):
+    def run(self) -> None:
+        super().run()
+        self.science_calibrate_frames()
+```
+
+**Bad example**:
+
+```python
+class ScienceCalibration(VispTaskBase):
+    def run(self) -> None:
+        self.science_calibrate_frames()
+```
+
+---
+
+## Task-Class Docstrings
+
+**Description**: Concrete task class docstrings are user-facing narrative documentation.
+
+**Flag when**:
+
+- A new or changed concrete task class lacks a docstring.
+- The docstring is only a developer note and does not explain what the task does for pipeline users/operators.
+- The docstring repeats the class name without adding useful context.
+
+**Good example**:
+
+```python
+class DarkCalibration(VispTaskBase):
+    """Create dark calibration arrays for downstream ViSP calibration tasks."""
+```
+
+**Bad example**:
+
+```python
+class DarkCalibration(VispTaskBase):
+    """DarkCalibration class."""
+```
+
+---
+
+## Tracing Span Usage
+
+**Description**: Tracing spans using `telemetry_span...` should provide useful operational visibility without overwhelming telemetry.
+
+**Flag when**:
+
+- A span name is vague, e.g. `run`, `process`, `loop`, `do stuff`, or merely repeats the task name.
+- A span is created inside an unbounded loop such as raster steps, DSPS repeats, frames of arbitrary count, or input records without a fixed upper bound.
+- A single added span mirrors the always-on run/task span but provides no additional visibility.
+- Load, process, and write phases are mixed together in one broad span when the code clearly separates those phases.
+- Span placement is so low in the loop structure that it risks producing excessive telemetry volume.
+
+**Acceptable pattern**:
+
+- Individual spans for bounded, known-small loop dimensions such as ViSP beam or VBI camera position.
+- Separate spans for load/process/write phases when they help diagnose runtime behavior.
+
+**Bad example**:
+
+```python
+for raster_step in raster_steps:
+    with telemetry_span(f"Process raster step {raster_step}"):
+        self.process_raster_step(raster_step)
+```
+
+---
+
+## FITS Header Access
+
+**Description**: FITS header names and values should be abstracted through `FitsAccessBase` and instrument-specific subclasses.
+
+**Flag when**:
+
+- Code outside FITS access classes reads FITS header keys directly from headers or dictionaries.
+- A raw Spec122 key is used where a translated `FitsAccessBase` attribute exists.
+  - Example: prefer `CAM_ID` over `CAM__001` when a translated attribute exists.
+- Header-version-specific logic is added outside FITS access classes/parsers.
+- Multiple modules duplicate the same raw header-key string.
+
+**Good example**:
+
+```python
+camera_id = fits_access.camera_id
+```
+
+**Bad example**:
+
+```python
+camera_id = header["CAM__001"]
+```
+
+---
+
+## Parameters and Parameter Storage
+
+**Description**: Parameter storage format should be chosen deliberately based on size, serialization needs, and operator readability.
+
+**Flag when**:
+
+- A new parameter stores large array-like data as JSON instead of a file-backed format such as `.npy`.
+- A small list or simple scalar is written as an opaque file when JSON would be clearer.
+- A file parameter is not self-describing enough to inspect with ordinary tools where practical.
+- The storage format appears to simply mirror the input format without considering downstream use.
+- The same parameter key/string is duplicated across multiple locations instead of being defined once.
+
+**Good pattern**:
+
+- Use JSON-compatible values for small scalar/list parameters.
+- Use `.npy` or another explicit file format for large array-like data.
+- Prefer named constants, enums, or parameter model attributes over repeated string literals.
+
+---
+
+## Logging Standards
+
+### Instrument Pipeline Repositories (`dkist-processing-[visp|vbi|...]`)
+Instrument-specific processing repositories should use the logging infrastructure provided by `dkist-service-configuration`.
+
+**Expected pattern**:
+```python
+from dkist_service_configuration.logging import logger
+```
+
+or the repository-standard wrapper around that logger.
+
+### Core and Shared Libraries (`dkist-processing-core`, `dkist-processing-common`)
+Core/shared libraries should use standard library logging with module-local loggers.
+
+**Expected pattern**:
+```python
+import logging
+
+logger = logging.getLogger(__name__)
+```
+
+### General Logging Rules
+- No `print()` statements in production code.
+- Rare exceptions may exist for notebook-oriented execution/debug workflows.
+- New modules should define a module-level logger rather than creating loggers inline repeatedly.
+- Logging should provide actionable operational/debugging information and avoid noisy per-item logging inside large loops unless rate-limited or scoped by tracing spans.
+
+---
+
+## Exceptions
+
+**Description**: Preserve exception context when converting or re-raising errors.
+
+**Flag when**:
+
+- Code catches an exception and raises a new exception without `raise ... from exc`.
+- Code catches broad exceptions without a clear recovery path.
+- Code logs an exception and continues in a way that may hide data loss or partial processing.
+
+**Good example**:
+
+```python
+try:
+    value = parse_value(raw_value)
+except ValueError as exc:
+    raise ParameterError("Could not parse parameter value") from exc
+```
+
+**Bad example**:
+
+```python
+try:
+    value = parse_value(raw_value)
+except ValueError:
+    raise ParameterError("Could not parse parameter value")
+```
+
+---
+
+## Type Hints
+
+**Description**: All new or modified functions and methods should have complete type hints.
+
+**Flag when**:
+
+- A function parameter lacks a type annotation.
+- A function lacks an explicit return type, including `-> None` for functions used only for side effects.
+- New code uses `Any`, untyped containers, or `dict`/`list` without parameterization where a concrete type is practical.
+
+**Good example**:
+
+```python
+def write_frame(frame_path: Path, tags: list[str]) -> None:
+    ...
+```
+
+**Bad example**:
+
+```python
+def write_frame(frame_path, tags):
+    ...
+```
+
+---
+
+## Avoid `**kwargs` for Core Behavior
+
+**Description**: Prefer explicit parameters over catch-all keyword arguments.
+
+**Flag when**:
+
+- New public functions, methods, or constructors use `**kwargs` without a strong reason.
+- `**kwargs` is used to pass required domain values such as task names, tags, parameter keys, FITS keys, or calibration settings.
+- `**kwargs` hides the contract between task classes and their collaborators.
+
+**Acceptable uses**:
+
+- Compatibility with third-party framework hooks that require `**kwargs`.
+- Thin adapters where the accepted keys are documented and validated immediately.
+
+---
+
+## Avoid Python Magic That Breaks Navigation
+
+**Description**: Avoid dynamic patterns that break IDE “Go To Definition” and “Where Used” for domain behavior.
+
+**Flag when**:
+
+- New code uses `getattr`, `setattr`, dynamic imports, monkeypatching, or computed method names for ordinary domain logic.
+- String names are used to reference methods/classes that could be imported or called directly.
+- Magic behavior makes task dependencies, parameter names, or FITS keys hard to trace.
+
+**Acceptable uses**:
+
+- Well-contained framework integration points.
+- Tests that intentionally patch or inspect behavior.
+
+---
+
+## Function and Method Size / Complexity
+
+**Description**: Large functions with multiple conceptual sections are hard to review, test, and maintain.
+
+**Flag when**:
+
+- A new or heavily modified function contains multiple comment-delimited sections such as “load”, “process”, and “write”.
+- A function introduces deep nested loops/conditionals that could be split into named helpers.
+- A long function mixes parsing, validation, processing, and output-writing logic.
+- Complex branching is driven by instrument/task type and would be clearer in separate task classes or polymorphic methods.
+
+**Good review suggestion**:
+
+> Consider extracting the commented `load`, `calibrate`, and `write` sections into named helper methods. That would make each phase easier to test and would give each block a docstring/narrative name.
+
+---
+
+## Names and Common Strings
+
+**Description**: Names should communicate domain meaning, and repeated strings should be centralized.
+
+**Flag when**:
+
+- Variables, methods, or functions have vague names such as `data`, `thing`, `stuff`, `foo`, `tmp`, or `val` in non-trivial code.
+- A commonly used string appears multiple times and should be represented by an enum, model attribute, constant, or tag class.
+- New task names, tags, metric codes, parameter names, or FITS-related strings are duplicated instead of added to the existing model/enum structure.
+
+---
+
+## Comments, TODOs, and Docstrings
+
+**Description**: Comments and docstrings should improve maintainability and generated documentation.
+
+**Flag when**:
+
+- New code adds TODO/FIXME comments without an issue reference, owner, or clear follow-up path.
+- A comment explains what a block of code does, but the block could be extracted into a well-named helper with a docstring.
+- A public function/method docstring is missing or too vague to help generated documentation.
+- Inline comments duplicate obvious code behavior instead of explaining intent or constraints.
+
+**Good example**:
+
+```python
+def calculate_beam_offsets(...) -> BeamOffsets:
+    """Calculate beam offsets used to align downstream ViSP calibration arrays."""
+```
+
+**Bad example**:
+
+```python
+# Calculate beam offsets
+beam_offsets = calculate_offsets(...)
+```
+
+---
+
+## Protected Members and Object Representation
+
+**Description**: Avoid reaching through object internals and ensure custom objects are debuggable.
+
+**Flag when**:
+
+- Production code accesses protected members such as `_value`, `_cache`, or `_internal_state` on objects it does not own.
+- New domain/model classes lack a useful `repr`/Pydantic representation/dataclass representation when instances are likely to appear in logs, debugging output, or assertion failures.
+
+---
+
+## Tests and Coverage
+
+**Description**: Changed behavior should be covered by targeted tests.
+
+**Flag when**:
+
+- New task behavior, parser behavior, FITS access behavior, parameter serialization, or workflow behavior is added without tests.
+- A bug fix lacks a regression test.
+- Tests assert only that code runs, without checking the meaningful output or side effect.
+- Coverage decreases in important domain logic without explanation.
+- Missing lines are justified only by “hard to test” when a smaller helper function would make the logic testable.
+
+**Acceptable explanation**:
+
+- Some branches require integration-test infrastructure or heavy mocking and are already covered by an integration workflow. In that case, ask for a PR note or test comment identifying the integration coverage.
+
+---
+
+## Notebook and Structured File Handling
+
+**Description**: Use native APIs for structured formats when accuracy matters.
+
+**Flag when**:
+
+- Notebook code manually reads text and then calls `nbformat.reads()` where `nbformat.read()` with a `Path` or file handle would be clearer.
+- Structured data such as JSON, notebooks, ASDF, or XML is searched as raw text when a native parser should be used.
+- A PR changes polling/performance-sensitive code to parse full structured files repeatedly without acknowledging the performance tradeoff.
+
+**Good example**:
+
+```python
+notebook = nbformat.read(notebook_path, as_version=nbformat.NO_CONVERT)
+```
+
+**Bad example**:
+
+```python
+notebook = nbformat.reads(notebook_path.read_text(encoding="utf-8"), as_version=nbformat.NO_CONVERT)
+```
+
+---
+
+## File Creation and Partial State
+
+**Description**: Validate inputs before creating output files to avoid partial state on errors.
+
+**Flag when**:
+
+- A function creates/touches output files before validating all inputs needed to populate those files.
+- A failure path can leave orphaned or partially written files without cleanup.
+- A write operation overwrites an existing output without an explicit overwrite policy.
+
+**Good pattern**:
+
+1. Collect inputs.
+2. Validate all required values.
+3. Build complete output content in memory or a temporary location.
+4. Write/touch the final output.
+
+---
+
+## Path Handling
+
+**Description**: Use `Path` operations consistently when working with paths.
+
+**Flag when**:
+
+- Code mixes string path manipulation with `pathlib.Path` objects.
+- Code uses built-in `open(path)` where `path` is already a `Path` object and `path.open(...)` would be more consistent with nearby code.
+- Code manually joins paths with string concatenation instead of `/` or `Path` methods.
+
+**Good example**:
+
+```python
+with output_path.open(mode="w", encoding="utf-8") as f:
+    f.write(contents)
+```
+
+**Bad example**:
+
+```python
+with open(output_path, "w") as f:
+    f.write(contents)
+```
+
+---
+
+## Review Comment Style
+
+When reporting an issue:
+
+- Be specific about the file/line and the convention being violated.
+- Explain the practical risk: broken dependency direction, hidden task coupling, loss of FITS-key abstraction, harder telemetry, partial file state, missing regression coverage, etc.
+- Suggest a concrete fix.
+- Do not ask for broad rewrites unless the changed code creates a clear maintainability or correctness risk.
+- Do not flag purely subjective naming/style issues unless the name obscures domain meaning or conflicts with existing conventions.
+
+**Preferred comment format**:
+
+```text
+This imports a concrete task from another task module. In dkist-processing-* task modules should import only from *_base modules or tasks.mixin so concrete workflow nodes do not become coupled to each other. Could this dependency move to a shared base/helper, or be expressed through the existing base class instead?
+```

{dkist_processing_visp-5.7.11 → dkist_processing_visp-5.7.13}/CHANGELOG.rst

@@ -1,3 +1,21 @@
+v5.7.13 (2026-05-19)
+====================
+
+Features
+--------
+
+- Update 'dkist-processing-common' to incorporate notebook export functionality for manually processed pipeline runs. (`#320 <https://bitbucket.org/dkistdc/dkist-processing-visp/pull-requests/320>`__)
+
+
+v5.7.12 (2026-05-18)
+====================
+
+Misc
+----
+
+- Make the `SubmitDatasetMetadata` task dependent on quality data assembly to prevent an issue with quality data causing orphaned dataset metadata. (`#313 <https://bitbucket.org/dkistdc/dkist-processing-visp/pull-requests/313>`__)
+
+
 v5.7.11 (2026-05-13)
 ====================