model-config-tests 0.2.2__tar.gz → 0.2.4__tar.gz

{model_config_tests-0.2.2 → model_config_tests-0.2.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: model_config_tests
-Version: 0.2.2
+Version: 0.2.4
 Summary: Test for ACCESS model (payu) configurations
 Author: ACCESS-NRI
 License: Apache-2.0
@@ -38,11 +38,19 @@ Code from these pytests is adapted from COSIMAS's ACCESS-OM2's [bit reproducibil
 
 ### How to run pytests manually on NCI
 
-1. Load payu module - this provides the dependencies needed to run the model
+1. Load payu module - this provides the dependencies needed to run the model.
 
     ```sh
     module use /g/data/vk83/modules
-    module load payu/1.1.6
+    module load payu
+    ```
+
+    Some model configurations may require a minimum payu version, specified in `config.yaml` as `payu_minimum_version`. Please ensure that your loaded payu module meets the requirement. 
+    If you need to run the model with a development version of payu, please use `payu/dev` instead:
+
+    ```sh
+    module use /g/data/vk83/modules
+    module load payu/dev
     ```
 
 2. Create and activate a python virtual environment for installing and running tests
@@ -52,10 +60,10 @@ Code from these pytests is adapted from COSIMAS's ACCESS-OM2's [bit reproducibil
     source <path/to/test-venv>/bin/activate
     ```
 
-3. Either pip install a released version of `model-config-tests`,
+3. Either pip install the latest released version of `model-config-tests`,
 
     ```sh
-    pip install model-config-tests==0.1.1
+    pip install model-config-tests
     ```
 
     Or to install `model-config-tests` in "editable" mode, first clone the repository, and then run pip install from the repository. This means any changes to the code are reflected in the installed package.
@@ -118,10 +126,15 @@ Running all tests in the pytest suite on a configuration will likely fail as the
 - `repro_determinism`: Determinism test that confirms repeated model runs give the same result.
 - `repro_determinism_restart`: Determinism test that confirms repeated experiments with two consecutive runs give the same result.
 - `repro_restart`: Restart reproducibility test that confirms two short consecutive model runs give the same result as a longer single model run.
+- `repro_payu_setup`: Test payu setup reproducibility; fail if MD5 of any file in manifest is changed.
+- `manifests_unchanged`: Uses `git diff` to check manifests are up-to-date. If only fast hashes (e.g. `binhash`) are different, the manifests are reproducible, but `payu setup` may take longer to run as `md5` hashes need to be recalculated.  This test is not intended for tagged configurations.
+- `manifests`: A shortcut to run both `manifests_unchanged` and `repro_payu_setup`.
 - `slow`: Tests that are slow to run
 - `dev_config`: General configuration QA tests.
 - `config`: Configuration QA tests for released branches. This includes the `dev_config` tests.
 
+
+
 There are also model-specific markers for configuration QA tests, e.g., `access_om2`, `access_esm1p5`, `access_om3` and `access_esm1p6`. For a list of all available markers,
 run:
 

{model_config_tests-0.2.2 → model_config_tests-0.2.4}/README.md

@@ -10,11 +10,19 @@ Code from these pytests is adapted from COSIMAS's ACCESS-OM2's [bit reproducibil
 
 ### How to run pytests manually on NCI
 
-1. Load payu module - this provides the dependencies needed to run the model
+1. Load payu module - this provides the dependencies needed to run the model.
 
     ```sh
     module use /g/data/vk83/modules
-    module load payu/1.1.6
+    module load payu
+    ```
+
+    Some model configurations may require a minimum payu version, specified in `config.yaml` as `payu_minimum_version`. Please ensure that your loaded payu module meets the requirement. 
+    If you need to run the model with a development version of payu, please use `payu/dev` instead:
+
+    ```sh
+    module use /g/data/vk83/modules
+    module load payu/dev
     ```
 
 2. Create and activate a python virtual environment for installing and running tests
@@ -24,10 +32,10 @@ Code from these pytests is adapted from COSIMAS's ACCESS-OM2's [bit reproducibil
     source <path/to/test-venv>/bin/activate
     ```
 
-3. Either pip install a released version of `model-config-tests`,
+3. Either pip install the latest released version of `model-config-tests`,
 
     ```sh
-    pip install model-config-tests==0.1.1
+    pip install model-config-tests
     ```
 
     Or to install `model-config-tests` in "editable" mode, first clone the repository, and then run pip install from the repository. This means any changes to the code are reflected in the installed package.
@@ -90,10 +98,15 @@ Running all tests in the pytest suite on a configuration will likely fail as the
 - `repro_determinism`: Determinism test that confirms repeated model runs give the same result.
 - `repro_determinism_restart`: Determinism test that confirms repeated experiments with two consecutive runs give the same result.
 - `repro_restart`: Restart reproducibility test that confirms two short consecutive model runs give the same result as a longer single model run.
+- `repro_payu_setup`: Test payu setup reproducibility; fail if MD5 of any file in manifest is changed.
+- `manifests_unchanged`: Uses `git diff` to check manifests are up-to-date. If only fast hashes (e.g. `binhash`) are different, the manifests are reproducible, but `payu setup` may take longer to run as `md5` hashes need to be recalculated.  This test is not intended for tagged configurations.
+- `manifests`: A shortcut to run both `manifests_unchanged` and `repro_payu_setup`.
 - `slow`: Tests that are slow to run
 - `dev_config`: General configuration QA tests.
 - `config`: Configuration QA tests for released branches. This includes the `dev_config` tests.
 
+
+
 There are also model-specific markers for configuration QA tests, e.g., `access_om2`, `access_esm1p5`, `access_om3` and `access_esm1p6`. For a list of all available markers,
 run:
 

{model_config_tests-0.2.2 → model_config_tests-0.2.4}/pyproject.toml

@@ -112,6 +112,8 @@ tag_prefix = "v"
 parentdir_prefix = "model_config_tests-"
 
 [tool.coverage.run]
+patch = ["subprocess"]
 omit = [
-    "src/model_config_tests/_version.py"
+    "*/model_config_tests/_version.py",
+    "src/model_config_tests/_version.py",
 ]

{model_config_tests-0.2.2 → model_config_tests-0.2.4}/src/model_config_tests/_version.py

@@ -8,11 +8,11 @@ import json
 
 version_json = '''
 {
- "date": "2025-10-21T07:48:35+1100",
+ "date": "2026-06-02T16:10:30+1000",
  "dirty": false,
  "error": null,
- "full-revisionid": "5b34e0c7b8cb513b5204a8baa5f97b22f84c2407",
- "version": "0.2.2"
+ "full-revisionid": "ee4cb743816648753c36e15e6a630bedc2859853",
+ "version": "0.2.4"
 }
 '''  # END VERSION_JSON
 

{model_config_tests-0.2.2 → model_config_tests-0.2.4}/src/model_config_tests/config_tests/conftest.py

@@ -131,6 +131,18 @@ def pytest_configure(config):
         "markers",
         "repro_determinism_restart: mark tests that check determinism restart",
     )
+    config.addinivalue_line(
+        "markers",
+        "repro_payu_setup: mark tests that check payu setup reproducibility",
+    )
+    config.addinivalue_line(
+        "markers",
+        "manifests: mark tests that check payu setup does not change manifests files or md5",
+    )
+    config.addinivalue_line(
+        "markers",
+        "manifests_unchanged: mark tests that check payu setup does not change manifests files",
+    )
     config.addinivalue_line("markers", "slow: mark tests that are slow to run")
     config.addinivalue_line(
         "markers",

{model_config_tests-0.2.2 → model_config_tests-0.2.4}/src/model_config_tests/config_tests/qa/test_access_esm1p6_config.py

@@ -27,8 +27,9 @@ ACCESS_ESM1P6_REPOSITORY_NAME = "ACCESS-ESM1.6"
 VALID_REALMS: set[str] = {"atmos", "land", "ocean", "ocnBgchem", "seaIce"}
 VALID_KEYWORDS: set[str] = {"global", "access-esm1.6"}
 VALID_NOMINAL_RESOLUTION: str = "100 km"
-# TODO: Add back in when valid DOI for ESM1.6 is obtained
-# VALID_REFERENCE: str = "https://doi.org/10.1071/ES19035"
+# TODO: Update this reference when ESM1.6 paper is ready
+VALID_REFERENCE_1p6: str = "https://doi.org/10.5281/zenodo.17490072"
+VALID_URL: str = "https://github.com/ACCESS-NRI/access-esm1.6-configs.git"
 VALID_RUNTIME: dict[str, int] = {"years": 1, "months": 0, "days": 0}
 VALID_RESTART_FREQ: str = "10YS"
 VALID_MPPNCCOMBINE_EXE: str = "mppnccombine.spack"
@@ -150,8 +151,9 @@ class TestAccessEsm1p6:
         "field,expected",
         [
             ("nominal_resolution", VALID_NOMINAL_RESOLUTION),
-            # TODO: Add back in when valid DOI for ESM1.6 is obtained (see commented constant above)
-            # ("reference", VALID_REFERENCE),
+            ("model", ACCESS_ESM1P6_REPOSITORY_NAME),
+            ("url", VALID_URL),
+            ("reference", VALID_REFERENCE_1p6),
         ],
     )
     def test_metadata_field_equal_expected_value(self, field, expected, metadata):
@@ -159,6 +161,19 @@ class TestAccessEsm1p6:
             field, "metadata.yaml", expected
         )
 
+    @pytest.mark.parametrize(
+        "field",
+        [
+            ("description"),
+            ("notes"),
+        ],
+    )
+    def test_metadata_not_contain_esm1p5(self, field, metadata):
+        """Check that some fields in metadata do not contain 'ESM1.5', e.g., notes and description."""
+        assert (
+            field in metadata and "ESM1.5" not in metadata[field]
+        ), f"Field '{field}' in metadata.yaml should not contain 'ESM1.5'. "
+
     def test_config_runtime(self, config):
         assert (
             "calendar" in config

{model_config_tests-0.2.2 → model_config_tests-0.2.4}/src/model_config_tests/config_tests/qa/test_access_om2_config.py

@@ -66,10 +66,13 @@ class AccessOM2Branch:
         self.set_resolution()
 
         self.is_high_resolution = self.resolution in ["025deg", "01deg"]
-        self.is_bgc = "bgc" in branch_name
+        is_bgc_old = "bgc" in branch_name
+        is_bgc_new = "wombat" in branch_name
+        self.is_bgc = is_bgc_old or is_bgc_new
 
         # Set expected module and model repository names
-        if self.is_bgc:
+        if is_bgc_old:
+            # Pre-generic-tracers BGC uses a separate exe
             self.module_name = ACCESS_OM2_BGC_MODULE_NAME
             self.model_repository_name = ACCESS_OM2_BGC_REPOSITORY_NAME
         else:
@@ -116,7 +119,7 @@ class TestAccessOM2:
 
     def test_mppncombine_fast_collate_exe(self, config, branch):
         if branch.is_high_resolution:
-            pattern = r"/g/data/vk83/apps/mppnccombine-fast/.*/bin/mppnccombine-fast"
+            pattern = r".*mppnccombine-fast"
             if "collate" in config:
                 assert re.match(
                     pattern, config["collate"]["exe"]
@@ -126,19 +129,6 @@ class TestAccessOM2:
                     "mpi"
                 ], "Expect `mpi: true` when using mppnccombine-fast"
 
-    def test_sync_userscript_ice_concatenation(self, config):
-        # This script runs in the sync pbs job before syncing output to a
-        # remote location
-        script = "/g/data/vk83/apps/om2-scripts/concatenate_ice/concat_ice_daily.sh"
-        assert (
-            "userscripts" in config
-            and "sync" in config["userscripts"]
-            and config["userscripts"]["sync"] == script
-        ), (
-            "Expect sync userscript set to ice-concatenation script."
-            + f"\nuserscript:\n  sync: {script}"
-        )
-
     def test_metadata_realm(self, metadata, branch):
         expected_realms = {"ocean", "seaIce"}
         expected_config = "realm:\n - ocean\n - seaIce"

{model_config_tests-0.2.2 → model_config_tests-0.2.4}/src/model_config_tests/config_tests/qa/test_config.py

@@ -225,11 +225,18 @@ class TestConfig:
                 "enable"
             ], "Sync to remote archive should not be enabled"
 
-    def test_sync_path_is_not_set(self, config):
+    def test_sync_base_path_is_not_set(self, config):
         if "sync" in config:
             assert not (
-                "path" in config["sync"] and config["sync"]["path"] is not None
-            ), "Sync path to remote archive should not be set"
+                "base_path" in config["sync"]
+                and config["sync"]["base_path"] is not None
+            ), "Sync base path to remote archive should not be configured"
+
+    def test_sync_path_not_exists(self, config):
+        if "sync" in config:
+            assert (
+                "path" not in config["sync"]
+            ), "Sync path should not exist since base_path is preferred"
 
     def test_experiment_name_is_not_defined(self, config):
         assert "experiment" not in config, (

{model_config_tests-0.2.2 → model_config_tests-0.2.4}/src/model_config_tests/config_tests/test_bit_reproducibility.py

@@ -9,7 +9,7 @@ from typing import Optional
 
 import pytest
 
-from model_config_tests.exp_test_helper import Experiments
+from model_config_tests.exp_test_helper import Experiments, ExpTestHelper, setup_exp
 from model_config_tests.util import DAY_IN_SECONDS, HOUR_IN_SECONDS
 
 # Names of shared experiments
@@ -147,6 +147,20 @@ def experiments(
     return _experiments(experiments_markers, output_path, control_path, keep_archive)
 
 
+@pytest.fixture
+def requested_experiments(request, experiments: Experiments):
+    """Fixture to check that requested experiments have run successfully
+    and return a dictionary of ExpTestHelper instances for each experiment."""
+    exp_marker = request.node.get_closest_marker("experiments").args[0]
+    requested_exps = {}
+    for exp_name in exp_marker:
+        # Check experiment has run successfully - this will raise an
+        # error if there are any non-zero exit codes in the outputs
+        experiments.check_experiment(exp_name)
+        requested_exps[exp_name] = experiments.get_experiment(exp_name)
+    return requested_exps
+
+
 class TestBitReproducibility:
 
     @pytest.mark.repro
@@ -160,7 +174,7 @@ class TestBitReproducibility:
         self,
         output_path: Path,
         control_path: Path,
-        experiments: Experiments,
+        requested_experiments: dict[str, ExpTestHelper],
         checksum_path: Optional[Path],
     ):
         """
@@ -178,9 +192,9 @@ class TestBitReproducibility:
             Path to the model configuration to test. This is copied for
             for control directories in experiments. Default is set in
             conftests.py.
-        experiments: Experiments
-            Class that manages the shared experiments. This is a fixture
-            defined in this file.
+        requested_experiments: dict[str, ExpTestHelper]
+            A dictionary of requested experiments, where the key is the
+            experiment name and the value is an instance of ExpTestHelper.
         checksum_path: Optional[Path]
             Path to checksums to compare model output against. Default is
             set to checksums saved on model configuration. This is a
@@ -190,12 +204,7 @@ class TestBitReproducibility:
         checksum_output_dir = set_checksum_output_dir(output_path=output_path)
 
         # Use default runtime experiment to get the historical checksums
-        experiments.check_experiments([EXP_DEFAULT_RUNTIME])
-        exp = experiments.get_experiment(EXP_DEFAULT_RUNTIME)
-
-        assert (
-            exp.model.output_exists()
-        ), "Output file required for model checksums does not exist"
+        exp = requested_experiments.get(EXP_DEFAULT_RUNTIME)
 
         # Set the checksum output filename using the model default runtime
         runtime_hours = exp.model.default_runtime_seconds // HOUR_IN_SECONDS
@@ -235,20 +244,16 @@ class TestBitReproducibility:
             EXP_1D_RUNTIME_REPEAT: {"n_runs": 1, "model_runtime": DAY_IN_SECONDS},
         }
     )
-    def test_repro_determinism(self, experiments: Experiments):
+    def test_repro_determinism(self, requested_experiments: dict[str, ExpTestHelper]):
         """
         Determinism test that confirms repeated model runs for 1 day
         give the same results
         """
-        experiments.check_experiments([EXP_1D_RUNTIME, EXP_1D_RUNTIME_REPEAT])
-        exp_1d_runtime = experiments.get_experiment(EXP_1D_RUNTIME)
-        exp_1d_runtime_repeat = experiments.get_experiment(EXP_1D_RUNTIME_REPEAT)
+        exp_1d_runtime = requested_experiments.get(EXP_1D_RUNTIME)
+        exp_1d_runtime_repeat = requested_experiments.get(EXP_1D_RUNTIME_REPEAT)
 
         # Compare expected to produced.
-        assert exp_1d_runtime.model.output_exists()
         expected = exp_1d_runtime.extract_checksums()
-
-        assert exp_1d_runtime_repeat.model.output_exists()
         produced = exp_1d_runtime_repeat.extract_checksums()
 
         assert produced == expected
@@ -262,16 +267,17 @@ class TestBitReproducibility:
             EXP_2D_RUNTIME: {"n_runs": 1, "model_runtime": 2 * DAY_IN_SECONDS},
         }
     )
-    def test_repro_restart(self, output_path: Path, experiments: Experiments):
+    def test_repro_restart(
+        self, output_path: Path, requested_experiments: dict[str, ExpTestHelper]
+    ):
         """
         Restart reproducibility test that confirms two short consecutive
         1-day model runs give the same results as a longer single 2-day model
         run.
         """
         # Get experiments with 2x1 day and 2 day runtimes
-        experiments.check_experiments([EXP_1D_RUNTIME, EXP_2D_RUNTIME])
-        exp_1d_runtime = experiments.get_experiment(EXP_1D_RUNTIME)
-        exp_2d_runtime = experiments.get_experiment(EXP_2D_RUNTIME)
+        exp_1d_runtime = requested_experiments.get(EXP_1D_RUNTIME)
+        exp_2d_runtime = requested_experiments.get(EXP_2D_RUNTIME)
 
         # Now compare the output between our two short and one long run.
         checksums_1d_0 = exp_1d_runtime.extract_checksums()
@@ -305,14 +311,15 @@ class TestBitReproducibility:
             EXP_1D_RUNTIME_REPEAT: {"n_runs": 2, "model_runtime": DAY_IN_SECONDS},
         }
     )
-    def test_repro_determinism_restart(self, experiments: Experiments):
+    def test_repro_determinism_restart(
+        self, requested_experiments: dict[str, ExpTestHelper]
+    ):
         """
         Determinism test that confirms repeated experiments with two
         consecutive 1-day model runs give the same results
         """
-        experiments.check_experiments([EXP_1D_RUNTIME, EXP_1D_RUNTIME_REPEAT])
-        exp_1d_runtime = experiments.get_experiment(EXP_1D_RUNTIME)
-        exp_1d_runtime_repeat = experiments.get_experiment(EXP_1D_RUNTIME_REPEAT)
+        exp_1d_runtime = requested_experiments.get(EXP_1D_RUNTIME)
+        exp_1d_runtime_repeat = requested_experiments.get(EXP_1D_RUNTIME_REPEAT)
 
         # Extract checksums, using the output from the second model run
         expected = exp_1d_runtime.extract_checksums(exp_1d_runtime.model.output_1)
@@ -321,3 +328,32 @@ class TestBitReproducibility:
         )
 
         assert produced == expected
+
+
+@pytest.mark.repro
+@pytest.mark.manifests
+@pytest.mark.repro_payu_setup
+def test_repro_payu_setup(control_path, output_path):
+    """
+    Test payu setup with `--repro` flag which errors if md5 of any files in payu manifests are changed.
+    """
+    experiment = setup_exp(control_path, output_path, exp_name="repro_payu_setup")
+    try:
+        experiment.setup_reproduce()
+    except Exception as error:
+        pytest.fail(f"{error}")
+
+
+@pytest.mark.manifests
+@pytest.mark.manifests_unchanged
+def test_manifests_unchanged(control_path, output_path):
+    """
+    Test payu setup with `git diff` which errors if any files in payu manifests are changed.
+    """
+    experiment = setup_exp(
+        control_path, output_path, exp_name="setup_unchanged_manifests"
+    )
+    try:
+        experiment.setup_manifests_unchanged()
+    except Exception as error:
+        pytest.fail(f"{error}")

{model_config_tests-0.2.2 → model_config_tests-0.2.4}/src/model_config_tests/exp_test_helper.py

@@ -82,6 +82,93 @@ class ExpTestHelper:
         """
         return self.model.output_exists()
 
+    def setup(self, reproduce=False):
+        """
+        Run payu setup command. If reproduce is True, run with --reproduce flag
+        to check if md5 hashes have changed in the manifests.
+        """
+        owd = Path.cwd()
+        # Change to experiment directory and run.
+        os.chdir(self.control_path)
+
+        try:
+            setup_command = [
+                "payu",
+                "setup",
+                "--lab",
+                str(self.lab_path),
+            ]
+            if reproduce:
+                setup_command.append("--reproduce")
+            print(f"Running payu setup command: {setup_command}")
+            result = sp.run(setup_command, capture_output=True, text=True)
+
+        finally:
+            # Change back to original working directory
+            os.chdir(owd)
+
+        if result.returncode != 0:
+            raise RuntimeError(
+                "Failed to run payu setup"
+                + (" with --reproduce.\n" if reproduce else ".\n")
+                + f"{'='*10}STDOUT{'='*10}\n {result.stdout}\n"
+                f"{'='*10}STDERR{'='*10}\n {result.stderr}\n"
+            )
+
+    def run_git_diff(self, path, extra_args=None):
+        """
+        Run git diff command on the given path and return the output.
+        """
+        command = ["git", "-C", str(path), "diff"] + extra_args if extra_args else []
+
+        result = sp.run(command, capture_output=True, text=True)
+
+        if result.returncode != 0:
+            raise RuntimeError(
+                f"Git command failed with exit code {result.returncode}.\n"
+                f"{'='*10}STDOUT{'='*10}\n {result.stdout}\n"
+                f"{'='*10}STDERR{'='*10}\n {result.stderr}\n"
+            )
+
+        return result.stdout
+
+    def setup_reproduce(self):
+        """
+        Run payu setup with `--repro` flag to check if md5 hashes have changed in the manifests.
+        """
+        self.setup(reproduce=True)
+
+    def setup_manifests_unchanged(self):
+        """
+        Run payu setup command and check if manifests files have been changed with `git diff`.
+        """
+        self.setup(reproduce=False)
+
+        result = self.run_git_diff(
+            self.control_path, extra_args=["--name-only", "manifests/"]
+        )
+        if result != "":
+            # Collect and display the top 10 lines of the diff for each modified file
+            files = result.strip().split("\n")
+            error_message = "Modifications are detected in file:\n"
+            error_message += "\n".join(" - " + file for file in files) + "\n"
+            error_message += "\nIf md5 hashes have changed, this indicates file contents being different."
+            error_message += """
+If binhashes/paths have changed but md5's are the same,
+this will mean the configuration can reproduce the manifests
+but `payu setup` will take longer to run as it needs to re-calculate all the md5 hashes.
+            """
+            for file in files:
+                diff_details = self.run_git_diff(
+                    self.control_path, extra_args=[f"{file}"]
+                )
+                diff_lines = diff_details.splitlines()
+                top_lines = "\n".join(diff_lines[2:12])
+                if len(diff_lines) > 12:
+                    top_lines += "\n... (truncated)"
+                error_message += f"\n{'='*10} Diff for {file} {'='*10}\n{top_lines}\n"
+            raise RuntimeError(f"{error_message}")
+
     def setup_for_test_run(self):
         """
         Various config.yaml settings need to be modified in order to run in the
@@ -136,9 +223,30 @@ class ExpTestHelper:
             # Change to experiment directory and run.
             os.chdir(self.control_path)
 
-            print("Running payu setup and payu sweep commands")
-            sp.run(["payu", "setup", "--lab", str(self.lab_path)], check=True)
-            sp.run(["payu", "sweep", "--lab", str(self.lab_path)], check=True)
+            print("Running payu setup")
+            result = sp.run(
+                ["payu", "setup", "--lab", str(self.lab_path)],
+                capture_output=True,
+                text=True,
+            )
+            if result.returncode != 0:
+                # Add additional error messaging for debugging
+                error_msg = (
+                    "Failed to run payu setup:\n"
+                    f"Return code: {result.returncode}\n"
+                    f"--- stdout ---\n{result.stdout}\n"
+                    f"--- stderr ---\n{result.stderr}"
+                )
+                print(error_msg)
+                raise RuntimeError(error_msg)
+
+            print("Running payu sweep")
+            sp.run(
+                ["payu", "sweep", "--lab", str(self.lab_path)],
+                capture_output=True,
+                text=True,
+                check=True,
+            )
 
             run_command = ["payu", "run", "--lab", str(self.lab_path)]
             if n_runs:
@@ -208,7 +316,7 @@ class Experiments:
         self.output_path = output_path
         self.keep_archive = keep_archive
         self.experiments = {}
-        self.successful_experiments = []
+        self.experiment_errors = {}
 
     def setup_and_submit(
         self,
@@ -282,22 +390,27 @@ class Experiments:
             try:
                 exp.wait_for_payu_run()
                 print(f"Experiment {exp_name} completed successfully")
-                self.successful_experiments.append(exp_name)
             except RuntimeError as e:
+                self.experiment_errors[exp_name] = str(e)
                 if catch_errors:
-                    print(f"Error in experiment {exp_name}: {e}")
+                    print(f"Error running experiment {exp_name}: {e}")
                 else:
-                    raise e
+                    raise
 
-    def check_experiments(self, exp_names=list[str]) -> None:
+    def check_experiment(self, exp_name: str) -> None:
         """
-        Check whether given experiments names have run successfully
+        Check whether given experiment name has run successfully
         """
-        for exp_name in exp_names:
-            # TODO: Is there other useful information to display here?
-            assert (
-                exp_name in self.successful_experiments
-            ), f"There was an error running experiment: {exp_name}"
+        if exp_name in self.experiment_errors:
+            raise RuntimeError(
+                f"There was an error running experiment {exp_name}:"
+                f" {self.experiment_errors[exp_name]}"
+            )
+
+        # Double check if the required experiment output exists
+        exp = self.experiments.get(exp_name)
+        if not exp.model.output_exists():
+            raise RuntimeError(f"Experiment {exp_name} output file does not exist.")
 
 
 def setup_exp(
@@ -519,13 +632,13 @@ def wait_for_qsub_job(
     # Check whether the run job was successful
     exit_status = parse_exit_status_from_file(stdout)
     if exit_status != 0:
-        print(
+        raise RuntimeError(
+            f"Payu {job_type} job failed with exit status {exit_status}:\n"
             f"Job_ID: {job_id}\n"
             f"Output files: {output_files}\n"
             f"--- stdout ---\n{stdout}\n"
             f"--- stderr ---\n{stderr}\n"
         )
-        raise RuntimeError(f"Payu {job_type} job failed with exit status {exit_status}")
 
     return stdout, stderr, output_files
 

{model_config_tests-0.2.2 → model_config_tests-0.2.4}/src/model_config_tests.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: model_config_tests
-Version: 0.2.2
+Version: 0.2.4
 Summary: Test for ACCESS model (payu) configurations
 Author: ACCESS-NRI
 License: Apache-2.0
@@ -38,11 +38,19 @@ Code from these pytests is adapted from COSIMAS's ACCESS-OM2's [bit reproducibil
 
 ### How to run pytests manually on NCI
 
-1. Load payu module - this provides the dependencies needed to run the model
+1. Load payu module - this provides the dependencies needed to run the model.
 
     ```sh
     module use /g/data/vk83/modules
-    module load payu/1.1.6
+    module load payu
+    ```
+
+    Some model configurations may require a minimum payu version, specified in `config.yaml` as `payu_minimum_version`. Please ensure that your loaded payu module meets the requirement. 
+    If you need to run the model with a development version of payu, please use `payu/dev` instead:
+
+    ```sh
+    module use /g/data/vk83/modules
+    module load payu/dev
     ```
 
 2. Create and activate a python virtual environment for installing and running tests
@@ -52,10 +60,10 @@ Code from these pytests is adapted from COSIMAS's ACCESS-OM2's [bit reproducibil
     source <path/to/test-venv>/bin/activate
     ```
 
-3. Either pip install a released version of `model-config-tests`,
+3. Either pip install the latest released version of `model-config-tests`,
 
     ```sh
-    pip install model-config-tests==0.1.1
+    pip install model-config-tests
     ```
 
     Or to install `model-config-tests` in "editable" mode, first clone the repository, and then run pip install from the repository. This means any changes to the code are reflected in the installed package.
@@ -118,10 +126,15 @@ Running all tests in the pytest suite on a configuration will likely fail as the
 - `repro_determinism`: Determinism test that confirms repeated model runs give the same result.
 - `repro_determinism_restart`: Determinism test that confirms repeated experiments with two consecutive runs give the same result.
 - `repro_restart`: Restart reproducibility test that confirms two short consecutive model runs give the same result as a longer single model run.
+- `repro_payu_setup`: Test payu setup reproducibility; fail if MD5 of any file in manifest is changed.
+- `manifests_unchanged`: Uses `git diff` to check manifests are up-to-date. If only fast hashes (e.g. `binhash`) are different, the manifests are reproducible, but `payu setup` may take longer to run as `md5` hashes need to be recalculated.  This test is not intended for tagged configurations.
+- `manifests`: A shortcut to run both `manifests_unchanged` and `repro_payu_setup`.
 - `slow`: Tests that are slow to run
 - `dev_config`: General configuration QA tests.
 - `config`: Configuration QA tests for released branches. This includes the `dev_config` tests.
 
+
+
 There are also model-specific markers for configuration QA tests, e.g., `access_om2`, `access_esm1p5`, `access_om3` and `access_esm1p6`. For a list of all available markers,
 run:
 

{model_config_tests-0.2.2 → model_config_tests-0.2.4}/tests/test_exp_test_helper.py

@@ -1,13 +1,14 @@
 import shutil
 import subprocess
 from pathlib import Path
-from unittest.mock import patch
+from unittest.mock import MagicMock, Mock, patch
 
 import pytest
 import yaml
 from netCDF4 import Dataset
 
 from model_config_tests.exp_test_helper import (
+    Experiments,
     ExpTestHelper,
     parse_exit_status_from_file,
     parse_gadi_pbs_ids,
@@ -127,6 +128,7 @@ def test_experiment_setup_for_test_run_remove_postprocessing(exp, tmp_path):
 @patch("subprocess.run")
 def test_experiment_submit_payu_run(mock_run, exp):
     mock_run.return_value.stdout = "1234567.gadi-pbs\nsome other output"
+    mock_run.return_value.returncode = 0
 
     current_working_dir = Path.cwd()
     exp.submit_payu_run()
@@ -150,6 +152,7 @@ def test_experiment_submit_payu_run(mock_run, exp):
 def test_experiment_submit_payu_run_n_runs(mock_run, exp):
     """Test --n-runs is added to the payu run command"""
     mock_run.return_value.stdout = "1234567.gadi-pbs\nsome other output"
+    mock_run.return_value.returncode = 0
 
     exp.submit_payu_run(n_runs=2)
 
@@ -172,13 +175,33 @@ def test_experiment_submit_payu_run_disabled(mock_run, exp):
     assert job_id is None
 
 
+@patch("subprocess.run")
+def test_experiment_submit_payu_run_setup_error(mock_run, exp):
+    """Test that an error is raised when payu setup fails"""
+    mock_run.return_value.stdout = "Some output"
+    mock_run.return_value.stderr = "Some error"
+    mock_run.return_value.returncode = 1
+
+    with pytest.raises(RuntimeError, match="Failed to run payu setup*"):
+        exp.submit_payu_run()
+
+    assert exp.run_id is None
+
+
 @patch("subprocess.run")
 def test_experiment_submit_payu_run_error(mock_run, exp):
-    """Test that an error is raised when any payu command fails"""
-    mock_run.side_effect = subprocess.CalledProcessError(
-        returncode=1, cmd="payu setup", output="Some error"
-    )
-    mock_run.return_value.stdout = "Some error"
+    """Test that an RuntimeError is raised with CalledProcessError"""
+    # Mock the first call to payu setup to succeed
+    # and subsequent payu command to fail
+    setup_success = Mock()
+    setup_success.stdout = "Setup successful"
+    setup_success.returncode = 0
+    mock_run.side_effect = [
+        setup_success,
+        subprocess.CalledProcessError(
+            returncode=1, cmd="payu run", output="Some error"
+        ),
+    ]
 
     with pytest.raises(RuntimeError, match="Failed to submit payu run.*"):
         exp.submit_payu_run()
@@ -490,3 +513,136 @@ def test_extract_checksums_split_uses_first_tile(exp_with_restarts):
 
     checksums = exp_accessom3.extract_checksums(output_directory=exp_accessom3.output_0)
     assert checksums["output"]["DTBT"][0] == "AC87F8AC28BD1436"
+
+
+def test_experiments_check_experiment_error(tmp_path):
+    with patch("model_config_tests.exp_test_helper.setup_exp") as mock_setup_exp:
+        # Create an experiment that will error later on
+        mock_error_exp = Mock(autospec=ExpTestHelper)
+        mock_setup_exp.return_value = mock_error_exp
+        mock_error_exp.wait_for_payu_run.side_effect = RuntimeError(
+            "Payu run job failed with exit status 1"
+        )
+
+        exps = Experiments(
+            control_path=tmp_path / "control",
+            output_path=tmp_path / "output",
+            keep_archive=True,
+        )
+        exps.setup_and_submit(exp_name="error_exp")
+        assert exps.experiments["error_exp"] == mock_error_exp
+
+        # Add a second experiment that will succeed
+        mock_success_exp = Mock(autospec=ExpTestHelper)
+        mock_success_exp.wait_for_payu_run.return_value = None
+        mock_setup_exp.return_value = mock_success_exp
+
+        exps.setup_and_submit(exp_name="success_exp")
+        assert exps.experiments["success_exp"] == mock_success_exp
+
+    # Check no errors are raised here
+    exps.wait_for_all_experiments(catch_errors=True)
+    assert exps.experiment_errors == {
+        "error_exp": "Payu run job failed with exit status 1"
+    }
+
+    # Check no errors with successful experiment
+    exps.check_experiment("success_exp")
+
+    # Check error raised for the failed experiment
+    error_msg = (
+        "There was an error running experiment error_exp: "
+        "Payu run job failed with exit status 1"
+    )
+    with pytest.raises(RuntimeError, match=error_msg):
+        exps.check_experiment("error_exp")
+
+
+@patch("subprocess.run")
+def test_setup_reproduce_error(mock_run, exp):
+    """Test that payu setup --repro fails raises an error and return to original work directory"""
+    # Mock the payu setup --repro to fail
+    mock_result = MagicMock()
+    mock_result.returncode = 1
+    mock_result.stderr = "MD5 mismatch"
+    mock_result.stdout = "Check manifest"
+    mock_run.return_value = mock_result
+
+    # Store original current working directory
+    owd = Path.cwd()
+
+    with pytest.raises(RuntimeError) as excinfo:
+        exp.setup_reproduce()
+
+    assert "Failed to run payu setup with --reproduce.\n" in str(excinfo.value)
+    assert f"{'='*10}STDOUT{'='*10}\n {mock_result.stdout}\n" in str(excinfo.value)
+
+    # assert returning to the original work directory
+    assert Path.cwd() == owd
+
+
+@patch("subprocess.run")
+def test_setup_manifests_unchanged_fail_setup(mock_run, exp):
+    """Test that an error is raised when payu setup fails in setup_manifests_unchanged()"""
+    # Mock the payu setup --repro to fail with unchanged manifests
+    mock_result = MagicMock()
+    mock_result.returncode = 1
+    mock_result.stderr = "Setup failed"
+    mock_result.stdout = "Payu setup output"
+    mock_run.return_value = mock_result
+
+    # Store original current working directory
+    owd = Path.cwd()
+
+    with pytest.raises(RuntimeError) as excinfo:
+        exp.setup_manifests_unchanged()
+
+    assert "Failed to run payu setup" in str(excinfo.value)
+    assert f"{'='*10}STDOUT{'='*10}\n {mock_result.stdout}\n" in str(excinfo.value)
+
+    # assert returning to the original work directory
+    assert Path.cwd() == owd
+
+
+@patch("subprocess.run")
+def test_setup_manifests_unchanged_show_changes(mock_run, exp):
+    """Test that when manifests are changed, the `git diff` results are printed to stdout"""
+    # Mock the `payu setup` succeed first
+    setup_success = MagicMock(returncode=0, stdout="Payu setup succeeded")
+
+    top_lines = """--- a/{diff_file}
++++ b/{diff_file}
++new line
+-old line
+    """
+    diff_file = "manifests/input.yaml"
+    # Then mock the `git diff --name-only` to show which files are changed
+    git_diff_name_only = MagicMock(returncode=0, stdout=diff_file)
+
+    # Mock the `git diff` to show the detailed changes in the file
+    git_diff_run = MagicMock(
+        returncode=0,
+        stdout=(
+            f"""diff --git a/{diff_file} b/{diff_file}
+index abc123...zyx789 100111
+"""
+        )
+        + top_lines,
+    )
+
+    # Run these mocks in sequence
+    mock_run.side_effect = [setup_success, git_diff_name_only, git_diff_run]
+
+    # Store original current working directory
+    owd = Path.cwd()
+
+    with pytest.raises(RuntimeError) as excinfo:
+        exp.setup_manifests_unchanged()
+
+    assert "Modifications are detected in file:\n" in str(excinfo.value)
+    assert f"\n{'='*10} Diff for {diff_file} {'='*10}\n{top_lines}\n" in str(
+        excinfo.value
+    )
+
+    # assert returning to the original work directory
+    assert Path.cwd() == owd