PVNet 4.1.30__tar.gz → 5.0.0__tar.gz

{pvnet-4.1.30 → pvnet-5.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: PVNet
-Version: 4.1.30
+Version: 5.0.0
 Summary: PVNet
 Author-email: Peter Dudfield <info@openclimatefix.org>
 Requires-Python: >=3.10
@@ -15,8 +15,6 @@ Requires-Dist: h5netcdf
 Requires-Dist: torch>=2.0.0
 Requires-Dist: lightning
 Requires-Dist: torchvision
-Requires-Dist: pytest
-Requires-Dist: pytest-cov
 Requires-Dist: typer
 Requires-Dist: sqlalchemy
 Requires-Dist: fsspec[s3]
@@ -27,9 +25,10 @@ Requires-Dist: omegaconf
 Requires-Dist: hydra-core
 Requires-Dist: rich
 Requires-Dist: einops
+Requires-Dist: safetensors
 Dynamic: license-file
 
-# PVNet 2.1
+# PVNet
 <!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
 [![All Contributors](https://img.shields.io/badge/all_contributors-19-orange.svg?style=flat-square)](#contributors-)
 <!-- ALL-CONTRIBUTORS-BADGE:END -->
@@ -40,39 +39,34 @@ Dynamic: license-file
 
 This project is used for training PVNet and running PVNet on live data.
 
-PVNet2 is a multi-modal late-fusion model that largely inherits the same architecture from
-[PVNet1.0](https://github.com/openclimatefix/predict_pv_yield). The NWP (Numerical Weather Prediction) and
-satellite data are sent through some neural network which encodes them down to
-1D intermediate representations. These are concatenated together with the GSP (Grid Supply Point)
-output history, the calculated solar coordinates (azimuth and elevation) and the
-GSP ID which has been put through an embedding layer. This 1D concatenated
-feature vector is put through an output network which outputs predictions of the
-future GSP yield. National forecasts are made by adding all the GSP forecasts
-together.
+PVNet is a multi-modal late-fusion model for predicting renewable energy generation from weather 
+data. The NWP (Numerical Weather Prediction) and satellite data are sent through a neural network 
+which encodes them down to 1D intermediate representations. These are concatenated together with 
+recent generation, the calculated solar coordinates (azimuth and elevation) and the location ID 
+which has been put through an embedding layer. This 1D concatenated feature vector is put through 
+an output network which outputs predictions of the future energy yield.
 
 
 ## Experiments
 
-Our paper based on this repo was accepted into the Tackling Climate Change with Machine Learning workshop at ICLR 2024 and can be viewed [here](https://www.climatechange.ai/papers/iclr2024/46).
-
-Some slightly more structured notes on deliberate experiments we have performed with PVNet are [here](https://docs.google.com/document/d/1VumDwWd8YAfvXbOtJEv3ZJm_FHQDzrKXR0jU9vnvGQg).
-
-Some very rough, early working notes on this model are
-[here](https://docs.google.com/document/d/1fbkfkBzp16WbnCg7RDuRDvgzInA6XQu3xh4NCjV-WDA). These are now somewhat out of date.
+Our paper based on this repo was accepted into the Tackling Climate Change with Machine Learning 
+workshop at ICLR 2024 and can be viewed [here](https://www.climatechange.ai/papers/iclr2024/46).
 
+Some more structured notes on experiments we have performed with PVNet are 
+[here](https://docs.google.com/document/d/1VumDwWd8YAfvXbOtJEv3ZJm_FHQDzrKXR0jU9vnvGQg).
 
 
 ## Setup / Installation
 
 ```bash
-git clone https://github.com/openclimatefix/PVNet.git
+git clone git@github.com:openclimatefix/PVNet.git
 cd PVNet
 pip install .
 ```
 
 The commit history is extensive. To save download time, use a depth of 1:
 ```bash
-git clone --depth 1 https://github.com/openclimatefix/PVNet.git
+git clone --depth 1 git@github.com:openclimatefix/PVNet.git
 ```
 This means only the latest commit and its associated files will be downloaded.
 
@@ -130,7 +124,7 @@ here: https://huggingface.co/datasets/openclimatefix/uk_pv
 
 Outside the PVNet repo, clone the ocf-data-sampler repo and exit the conda env created for PVNet: https://github.com/openclimatefix/ocf-data-sampler
 ```bash
-git clone https://github.com/openclimatefix/ocf-data-sampler.git
+git clone git@github.com/openclimatefix/ocf-data-sampler.git
 conda create -n ocf-data-sampler python=3.11
 ```
 
@@ -146,7 +140,8 @@ Then exit this environment, and enter back into the pvnet conda environment and
 pip install -e <PATH-TO-ocf-data-sampler-REPO>
 ```
 
-If you install the local version of `ocf-data-sampler` that is more recent than the version specified in PVNet, you might receive a warning. However, it should still function correctly.
+If you install the local version of `ocf-data-sampler` that is more recent than the version 
+specified in `PVNet` it is not guarenteed to function properly with this library.
 
 ## Pre-saving samples of data for training/validation of PVNet
 
@@ -205,14 +200,14 @@ Files stored in multiple locations can be added as a list. For example, in the `
 
 ```yaml
 satellite:
-    satellite_zarr_path: gs://solar-pv-nowcasting-data/satellite/EUMETSAT/SEVIRI_RSS/v4/2020_nonhrv.zarr
+    zarr_path: gs://solar-pv-nowcasting-data/satellite/EUMETSAT/SEVIRI_RSS/v4/2020_nonhrv.zarr
 ```
 
 Or to satellite data hosted by Google:
 
 ```yaml
 satellite:
-    satellite_zarr_paths:
+    zarr_path:
       - "gs://public-datasets-eumetsat-solar-forecasting/satellite/EUMETSAT/SEVIRI_RSS/v4/2020_nonhrv.zarr"
       - "gs://public-datasets-eumetsat-solar-forecasting/satellite/EUMETSAT/SEVIRI_RSS/v4/2021_nonhrv.zarr"
 ```
@@ -227,13 +222,13 @@ files. The configs stored in `PVNet/configs.example` should work with samples cr
 
 Make sure to update the following config files before training your model:
 
-1. In `configs/datamodule/local_presaved_samples.yaml`:
+1. In `configs/datamodule/presaved_samples.yaml`:
     - update `sample_dir` to point to the directory you stored your samples in during sample creation
-2. In `configs/model/local_multimodal.yaml`:
+2. In `configs/model/late_fusion.yaml`:
     - update the list of encoders to reflect the data sources you are using. If you are using different NWP sources, the encoders for these should follow the same structure with two important updates:
         - `in_channels`: number of variables your NWP source supplies
         - `image_size_pixels`: spatial crop of your NWP data. It depends on the spatial resolution of your NWP; should match `image_size_pixels_height` and/or `image_size_pixels_width` in `datamodule/configuration/site_example_configuration.yaml` for the NWP, unless transformations such as coarsening was applied (e. g. as for ECMWF data)
-3. In `configs/local_trainer.yaml`:
+3. In `configs/trainer/default.yaml`:
     - set `accelerator: 0` if running on a system without a supported GPU
 
 If creating copies of the config files instead of modifying existing ones, update `defaults` in the main `./configs/config.yaml` file to use
@@ -241,11 +236,10 @@ your customised config files:
 
 ```yaml
 defaults:
-  - trainer: local_trainer.yaml
-  - model: local_multimodal.yaml
-  - datamodule: local_presaved_samples.yaml
+  - trainer: default.yaml
+  - model: late_fusion.yaml
+  - datamodule: presaved_samples.yaml
   - callbacks: null
-  - logger: csv.yaml
   - experiment: null
   - hparams_search: null
   - hydra: default.yaml

{pvnet-4.1.30 → pvnet-5.0.0}/PVNet.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: PVNet
-Version: 4.1.30
+Version: 5.0.0
 Summary: PVNet
 Author-email: Peter Dudfield <info@openclimatefix.org>
 Requires-Python: >=3.10
@@ -15,8 +15,6 @@ Requires-Dist: h5netcdf
 Requires-Dist: torch>=2.0.0
 Requires-Dist: lightning
 Requires-Dist: torchvision
-Requires-Dist: pytest
-Requires-Dist: pytest-cov
 Requires-Dist: typer
 Requires-Dist: sqlalchemy
 Requires-Dist: fsspec[s3]
@@ -27,9 +25,10 @@ Requires-Dist: omegaconf
 Requires-Dist: hydra-core
 Requires-Dist: rich
 Requires-Dist: einops
+Requires-Dist: safetensors
 Dynamic: license-file
 
-# PVNet 2.1
+# PVNet
 <!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
 [![All Contributors](https://img.shields.io/badge/all_contributors-19-orange.svg?style=flat-square)](#contributors-)
 <!-- ALL-CONTRIBUTORS-BADGE:END -->
@@ -40,39 +39,34 @@ Dynamic: license-file
 
 This project is used for training PVNet and running PVNet on live data.
 
-PVNet2 is a multi-modal late-fusion model that largely inherits the same architecture from
-[PVNet1.0](https://github.com/openclimatefix/predict_pv_yield). The NWP (Numerical Weather Prediction) and
-satellite data are sent through some neural network which encodes them down to
-1D intermediate representations. These are concatenated together with the GSP (Grid Supply Point)
-output history, the calculated solar coordinates (azimuth and elevation) and the
-GSP ID which has been put through an embedding layer. This 1D concatenated
-feature vector is put through an output network which outputs predictions of the
-future GSP yield. National forecasts are made by adding all the GSP forecasts
-together.
+PVNet is a multi-modal late-fusion model for predicting renewable energy generation from weather 
+data. The NWP (Numerical Weather Prediction) and satellite data are sent through a neural network 
+which encodes them down to 1D intermediate representations. These are concatenated together with 
+recent generation, the calculated solar coordinates (azimuth and elevation) and the location ID 
+which has been put through an embedding layer. This 1D concatenated feature vector is put through 
+an output network which outputs predictions of the future energy yield.
 
 
 ## Experiments
 
-Our paper based on this repo was accepted into the Tackling Climate Change with Machine Learning workshop at ICLR 2024 and can be viewed [here](https://www.climatechange.ai/papers/iclr2024/46).
-
-Some slightly more structured notes on deliberate experiments we have performed with PVNet are [here](https://docs.google.com/document/d/1VumDwWd8YAfvXbOtJEv3ZJm_FHQDzrKXR0jU9vnvGQg).
-
-Some very rough, early working notes on this model are
-[here](https://docs.google.com/document/d/1fbkfkBzp16WbnCg7RDuRDvgzInA6XQu3xh4NCjV-WDA). These are now somewhat out of date.
+Our paper based on this repo was accepted into the Tackling Climate Change with Machine Learning 
+workshop at ICLR 2024 and can be viewed [here](https://www.climatechange.ai/papers/iclr2024/46).
 
+Some more structured notes on experiments we have performed with PVNet are 
+[here](https://docs.google.com/document/d/1VumDwWd8YAfvXbOtJEv3ZJm_FHQDzrKXR0jU9vnvGQg).
 
 
 ## Setup / Installation
 
 ```bash
-git clone https://github.com/openclimatefix/PVNet.git
+git clone git@github.com:openclimatefix/PVNet.git
 cd PVNet
 pip install .
 ```
 
 The commit history is extensive. To save download time, use a depth of 1:
 ```bash
-git clone --depth 1 https://github.com/openclimatefix/PVNet.git
+git clone --depth 1 git@github.com:openclimatefix/PVNet.git
 ```
 This means only the latest commit and its associated files will be downloaded.
 
@@ -130,7 +124,7 @@ here: https://huggingface.co/datasets/openclimatefix/uk_pv
 
 Outside the PVNet repo, clone the ocf-data-sampler repo and exit the conda env created for PVNet: https://github.com/openclimatefix/ocf-data-sampler
 ```bash
-git clone https://github.com/openclimatefix/ocf-data-sampler.git
+git clone git@github.com/openclimatefix/ocf-data-sampler.git
 conda create -n ocf-data-sampler python=3.11
 ```
 
@@ -146,7 +140,8 @@ Then exit this environment, and enter back into the pvnet conda environment and
 pip install -e <PATH-TO-ocf-data-sampler-REPO>
 ```
 
-If you install the local version of `ocf-data-sampler` that is more recent than the version specified in PVNet, you might receive a warning. However, it should still function correctly.
+If you install the local version of `ocf-data-sampler` that is more recent than the version 
+specified in `PVNet` it is not guarenteed to function properly with this library.
 
 ## Pre-saving samples of data for training/validation of PVNet
 
@@ -205,14 +200,14 @@ Files stored in multiple locations can be added as a list. For example, in the `
 
 ```yaml
 satellite:
-    satellite_zarr_path: gs://solar-pv-nowcasting-data/satellite/EUMETSAT/SEVIRI_RSS/v4/2020_nonhrv.zarr
+    zarr_path: gs://solar-pv-nowcasting-data/satellite/EUMETSAT/SEVIRI_RSS/v4/2020_nonhrv.zarr
 ```
 
 Or to satellite data hosted by Google:
 
 ```yaml
 satellite:
-    satellite_zarr_paths:
+    zarr_path:
       - "gs://public-datasets-eumetsat-solar-forecasting/satellite/EUMETSAT/SEVIRI_RSS/v4/2020_nonhrv.zarr"
       - "gs://public-datasets-eumetsat-solar-forecasting/satellite/EUMETSAT/SEVIRI_RSS/v4/2021_nonhrv.zarr"
 ```
@@ -227,13 +222,13 @@ files. The configs stored in `PVNet/configs.example` should work with samples cr
 
 Make sure to update the following config files before training your model:
 
-1. In `configs/datamodule/local_presaved_samples.yaml`:
+1. In `configs/datamodule/presaved_samples.yaml`:
     - update `sample_dir` to point to the directory you stored your samples in during sample creation
-2. In `configs/model/local_multimodal.yaml`:
+2. In `configs/model/late_fusion.yaml`:
     - update the list of encoders to reflect the data sources you are using. If you are using different NWP sources, the encoders for these should follow the same structure with two important updates:
         - `in_channels`: number of variables your NWP source supplies
         - `image_size_pixels`: spatial crop of your NWP data. It depends on the spatial resolution of your NWP; should match `image_size_pixels_height` and/or `image_size_pixels_width` in `datamodule/configuration/site_example_configuration.yaml` for the NWP, unless transformations such as coarsening was applied (e. g. as for ECMWF data)
-3. In `configs/local_trainer.yaml`:
+3. In `configs/trainer/default.yaml`:
     - set `accelerator: 0` if running on a system without a supported GPU
 
 If creating copies of the config files instead of modifying existing ones, update `defaults` in the main `./configs/config.yaml` file to use
@@ -241,11 +236,10 @@ your customised config files:
 
 ```yaml
 defaults:
-  - trainer: local_trainer.yaml
-  - model: local_multimodal.yaml
-  - datamodule: local_presaved_samples.yaml
+  - trainer: default.yaml
+  - model: late_fusion.yaml
+  - datamodule: presaved_samples.yaml
   - callbacks: null
-  - logger: csv.yaml
   - experiment: null
   - hparams_search: null
   - hydra: default.yaml

pvnet-5.0.0/PVNet.egg-info/SOURCES.txt

@@ -0,0 +1,36 @@
+LICENSE
+README.md
+pyproject.toml
+PVNet.egg-info/PKG-INFO
+PVNet.egg-info/SOURCES.txt
+PVNet.egg-info/dependency_links.txt
+PVNet.egg-info/requires.txt
+PVNet.egg-info/top_level.txt
+pvnet/__init__.py
+pvnet/load_model.py
+pvnet/optimizers.py
+pvnet/utils.py
+pvnet/data/__init__.py
+pvnet/data/base_datamodule.py
+pvnet/data/site_datamodule.py
+pvnet/data/uk_regional_datamodule.py
+pvnet/models/__init__.py
+pvnet/models/base_model.py
+pvnet/models/ensemble.py
+pvnet/models/late_fusion/__init__.py
+pvnet/models/late_fusion/basic_blocks.py
+pvnet/models/late_fusion/late_fusion.py
+pvnet/models/late_fusion/encoders/__init__.py
+pvnet/models/late_fusion/encoders/basic_blocks.py
+pvnet/models/late_fusion/encoders/encoders3d.py
+pvnet/models/late_fusion/linear_networks/__init__.py
+pvnet/models/late_fusion/linear_networks/basic_blocks.py
+pvnet/models/late_fusion/linear_networks/networks.py
+pvnet/models/late_fusion/site_encoders/__init__.py
+pvnet/models/late_fusion/site_encoders/basic_blocks.py
+pvnet/models/late_fusion/site_encoders/encoders.py
+pvnet/training/__init__.py
+pvnet/training/lightning_module.py
+pvnet/training/plots.py
+pvnet/training/train.py
+tests/test_end2end.py

{pvnet-4.1.30 → pvnet-5.0.0}/PVNet.egg-info/requires.txt

@@ -7,8 +7,6 @@ h5netcdf
 torch>=2.0.0
 lightning
 torchvision
-pytest
-pytest-cov
 typer
 sqlalchemy
 fsspec[s3]
@@ -19,3 +17,4 @@ omegaconf
 hydra-core
 rich
 einops
+safetensors

{pvnet-4.1.30 → pvnet-5.0.0}/README.md

@@ -1,4 +1,4 @@
-# PVNet 2.1
+# PVNet
 <!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
 [![All Contributors](https://img.shields.io/badge/all_contributors-19-orange.svg?style=flat-square)](#contributors-)
 <!-- ALL-CONTRIBUTORS-BADGE:END -->
@@ -9,39 +9,34 @@
 
 This project is used for training PVNet and running PVNet on live data.
 
-PVNet2 is a multi-modal late-fusion model that largely inherits the same architecture from
-[PVNet1.0](https://github.com/openclimatefix/predict_pv_yield). The NWP (Numerical Weather Prediction) and
-satellite data are sent through some neural network which encodes them down to
-1D intermediate representations. These are concatenated together with the GSP (Grid Supply Point)
-output history, the calculated solar coordinates (azimuth and elevation) and the
-GSP ID which has been put through an embedding layer. This 1D concatenated
-feature vector is put through an output network which outputs predictions of the
-future GSP yield. National forecasts are made by adding all the GSP forecasts
-together.
+PVNet is a multi-modal late-fusion model for predicting renewable energy generation from weather 
+data. The NWP (Numerical Weather Prediction) and satellite data are sent through a neural network 
+which encodes them down to 1D intermediate representations. These are concatenated together with 
+recent generation, the calculated solar coordinates (azimuth and elevation) and the location ID 
+which has been put through an embedding layer. This 1D concatenated feature vector is put through 
+an output network which outputs predictions of the future energy yield.
 
 
 ## Experiments
 
-Our paper based on this repo was accepted into the Tackling Climate Change with Machine Learning workshop at ICLR 2024 and can be viewed [here](https://www.climatechange.ai/papers/iclr2024/46).
-
-Some slightly more structured notes on deliberate experiments we have performed with PVNet are [here](https://docs.google.com/document/d/1VumDwWd8YAfvXbOtJEv3ZJm_FHQDzrKXR0jU9vnvGQg).
-
-Some very rough, early working notes on this model are
-[here](https://docs.google.com/document/d/1fbkfkBzp16WbnCg7RDuRDvgzInA6XQu3xh4NCjV-WDA). These are now somewhat out of date.
+Our paper based on this repo was accepted into the Tackling Climate Change with Machine Learning 
+workshop at ICLR 2024 and can be viewed [here](https://www.climatechange.ai/papers/iclr2024/46).
 
+Some more structured notes on experiments we have performed with PVNet are 
+[here](https://docs.google.com/document/d/1VumDwWd8YAfvXbOtJEv3ZJm_FHQDzrKXR0jU9vnvGQg).
 
 
 ## Setup / Installation
 
 ```bash
-git clone https://github.com/openclimatefix/PVNet.git
+git clone git@github.com:openclimatefix/PVNet.git
 cd PVNet
 pip install .
 ```
 
 The commit history is extensive. To save download time, use a depth of 1:
 ```bash
-git clone --depth 1 https://github.com/openclimatefix/PVNet.git
+git clone --depth 1 git@github.com:openclimatefix/PVNet.git
 ```
 This means only the latest commit and its associated files will be downloaded.
 
@@ -99,7 +94,7 @@ here: https://huggingface.co/datasets/openclimatefix/uk_pv
 
 Outside the PVNet repo, clone the ocf-data-sampler repo and exit the conda env created for PVNet: https://github.com/openclimatefix/ocf-data-sampler
 ```bash
-git clone https://github.com/openclimatefix/ocf-data-sampler.git
+git clone git@github.com/openclimatefix/ocf-data-sampler.git
 conda create -n ocf-data-sampler python=3.11
 ```
 
@@ -115,7 +110,8 @@ Then exit this environment, and enter back into the pvnet conda environment and
 pip install -e <PATH-TO-ocf-data-sampler-REPO>
 ```
 
-If you install the local version of `ocf-data-sampler` that is more recent than the version specified in PVNet, you might receive a warning. However, it should still function correctly.
+If you install the local version of `ocf-data-sampler` that is more recent than the version 
+specified in `PVNet` it is not guarenteed to function properly with this library.
 
 ## Pre-saving samples of data for training/validation of PVNet
 
@@ -174,14 +170,14 @@ Files stored in multiple locations can be added as a list. For example, in the `
 
 ```yaml
 satellite:
-    satellite_zarr_path: gs://solar-pv-nowcasting-data/satellite/EUMETSAT/SEVIRI_RSS/v4/2020_nonhrv.zarr
+    zarr_path: gs://solar-pv-nowcasting-data/satellite/EUMETSAT/SEVIRI_RSS/v4/2020_nonhrv.zarr
 ```
 
 Or to satellite data hosted by Google:
 
 ```yaml
 satellite:
-    satellite_zarr_paths:
+    zarr_path:
       - "gs://public-datasets-eumetsat-solar-forecasting/satellite/EUMETSAT/SEVIRI_RSS/v4/2020_nonhrv.zarr"
       - "gs://public-datasets-eumetsat-solar-forecasting/satellite/EUMETSAT/SEVIRI_RSS/v4/2021_nonhrv.zarr"
 ```
@@ -196,13 +192,13 @@ files. The configs stored in `PVNet/configs.example` should work with samples cr
 
 Make sure to update the following config files before training your model:
 
-1. In `configs/datamodule/local_presaved_samples.yaml`:
+1. In `configs/datamodule/presaved_samples.yaml`:
     - update `sample_dir` to point to the directory you stored your samples in during sample creation
-2. In `configs/model/local_multimodal.yaml`:
+2. In `configs/model/late_fusion.yaml`:
     - update the list of encoders to reflect the data sources you are using. If you are using different NWP sources, the encoders for these should follow the same structure with two important updates:
         - `in_channels`: number of variables your NWP source supplies
         - `image_size_pixels`: spatial crop of your NWP data. It depends on the spatial resolution of your NWP; should match `image_size_pixels_height` and/or `image_size_pixels_width` in `datamodule/configuration/site_example_configuration.yaml` for the NWP, unless transformations such as coarsening was applied (e. g. as for ECMWF data)
-3. In `configs/local_trainer.yaml`:
+3. In `configs/trainer/default.yaml`:
     - set `accelerator: 0` if running on a system without a supported GPU
 
 If creating copies of the config files instead of modifying existing ones, update `defaults` in the main `./configs/config.yaml` file to use
@@ -210,11 +206,10 @@ your customised config files:
 
 ```yaml
 defaults:
-  - trainer: local_trainer.yaml
-  - model: local_multimodal.yaml
-  - datamodule: local_presaved_samples.yaml
+  - trainer: default.yaml
+  - model: late_fusion.yaml
+  - datamodule: presaved_samples.yaml
   - callbacks: null
-  - logger: csv.yaml
   - experiment: null
   - hparams_search: null
   - hydra: default.yaml

{pvnet-4.1.30 → pvnet-5.0.0}/pvnet/data/base_datamodule.py

@@ -5,16 +5,12 @@ from glob import glob
 import torch
 from lightning.pytorch import LightningDataModule
 from ocf_data_sampler.numpy_sample.collate import stack_np_samples_into_batch
-from ocf_data_sampler.torch_datasets.sample.base import (
-    NumpyBatch,
-    SampleBase,
-    TensorBatch,
-    batch_to_tensor,
-)
+from ocf_data_sampler.numpy_sample.common_types import NumpySample, TensorBatch
+from ocf_data_sampler.torch_datasets.sample.base import SampleBase, batch_to_tensor
 from torch.utils.data import DataLoader, Dataset, Subset
 
 
-def collate_fn(samples: list[NumpyBatch]) -> TensorBatch:
+def collate_fn(samples: list[NumpySample]) -> TensorBatch:
     """Convert a list of NumpySample samples to a tensor batch"""
     return batch_to_tensor(stack_np_samples_into_batch(samples))
 
@@ -32,10 +28,10 @@ class PresavedSamplesDataset(Dataset):
         self.sample_paths = glob(f"{sample_dir}/*")
         self.sample_class = sample_class
 
-    def __len__(self):
+    def __len__(self) -> int:
         return len(self.sample_paths)
 
-    def __getitem__(self, idx):
+    def __getitem__(self, idx) -> NumpySample:
         sample = self.sample_class.load(self.sample_paths[idx])
         return sample.to_numpy()
 

{pvnet-4.1.30 → pvnet-5.0.0}/pvnet/data/site_datamodule.py

@@ -15,8 +15,7 @@ class SitePresavedDataModule(BasePresavedDataModule):
     """Datamodule for loading pre-saved samples."""
 
     def _get_premade_samples_dataset(self, subdir: str) -> Dataset:
-        split_dir = f"{self.sample_dir}/{subdir}"
-        return PresavedSamplesDataset(split_dir, SiteSample)
+        return PresavedSamplesDataset(f"{self.sample_dir}/{subdir}", SiteSample)
 
 
 class SiteStreamedDataModule(BaseStreamedDataModule):

{pvnet-4.1.30 → pvnet-5.0.0}/pvnet/data/uk_regional_datamodule.py

@@ -15,8 +15,7 @@ class UKRegionalPresavedDataModule(BasePresavedDataModule):
     """Datamodule for loading pre-saved samples."""
 
     def _get_premade_samples_dataset(self, subdir: str) -> Dataset:
-        split_dir = f"{self.sample_dir}/{subdir}"
-        return PresavedSamplesDataset(split_dir, UKRegionalSample)
+        return PresavedSamplesDataset(f"{self.sample_dir}/{subdir}", UKRegionalSample)
 
 
 class UKRegionalStreamedDataModule(BaseStreamedDataModule):

{pvnet-4.1.30 → pvnet-5.0.0}/pvnet/load_model.py

@@ -2,17 +2,16 @@
 
 import glob
 import os
-from typing import Any
 
 import hydra
 import torch
-from pyaml_env import parse_config
+import yaml
 
 from pvnet.models.ensemble import Ensemble
-from pvnet.models.multimodal.unimodal_teacher import Model as UMTModel
 from pvnet.utils import (
     DATA_CONFIG_NAME,
     DATAMODULE_CONFIG_NAME,
+    FULL_CONFIG_NAME,
     MODEL_CONFIG_NAME,
 )
 
@@ -20,7 +19,7 @@ from pvnet.utils import (
 def get_model_from_checkpoints(
     checkpoint_dir_paths: list[str],
     val_best: bool = True,
-) -> tuple[torch.nn.Module, dict[str, Any] | str, str | None, str | None]:
+) -> tuple[torch.nn.Module, dict, str, str | None, str | None]:
     """Load a model from its checkpoint directory
 
     Returns:
@@ -29,6 +28,7 @@ def get_model_from_checkpoints(
             model_config: path to model config used to train the model.
             data_config: path to data config used to create samples for the model.
             datamodule_config: path to datamodule used to create samples e.g train/test split info.
+            experiment_configs: path to the full experimental config.
 
     """
     is_ensemble = len(checkpoint_dir_paths) > 1
@@ -37,12 +37,15 @@ def get_model_from_checkpoints(
     models = []
     data_configs = []
     datamodule_configs = []
+    experiment_configs = []
 
     for path in checkpoint_dir_paths:
-        # Load the model
-        model_config = parse_config(f"{path}/{MODEL_CONFIG_NAME}")
 
-        model = hydra.utils.instantiate(model_config)
+        # Load lightning training module
+        with open(f"{path}/{MODEL_CONFIG_NAME}") as cfg:
+            model_config = yaml.load(cfg, Loader=yaml.FullLoader)
+
+        lightning_module = hydra.utils.instantiate(model_config)
 
         if val_best:
             # Only one epoch (best) saved per model
@@ -52,33 +55,40 @@ def get_model_from_checkpoints(
                     f"Found {len(files)} checkpoints @ {path}/epoch*.ckpt. Expected one."
                 )
             # TODO: Loading with weights_only=False is not recommended
-            checkpoint = torch.load(files[0], map_location="cpu", weights_only=False)
+            checkpoint = torch.load(files[0], map_location="cpu", weights_only=True)
         else:
-            checkpoint = torch.load(f"{path}/last.ckpt", map_location="cpu", weights_only=False)
-
-        model.load_state_dict(state_dict=checkpoint["state_dict"])
+            checkpoint = torch.load(f"{path}/last.ckpt", map_location="cpu", weights_only=True)
 
-        if isinstance(model, UMTModel):
-            model, model_config = model.convert_to_multimodal_model(model_config)
+        lightning_module.load_state_dict(state_dict=checkpoint["state_dict"])
 
-        model_configs.append(model_config)
-        models.append(model)
+        # Extract the model from the lightning module
+        models.append(lightning_module.model)
+        model_configs.append(model_config["model"])
 
-        # Check for data config
+        # Store the data config used for the model
         data_config = f"{path}/{DATA_CONFIG_NAME}"
 
         if os.path.isfile(data_config):
             data_configs.append(data_config)
         else:
-            data_configs.append(None)
+            raise FileNotFoundError(f"File {data_config} does not exist")
 
-        # check for datamodule config
+        # Check for datamodule config
+        # This only exists if the model was trained with presaved samples
         datamodule_config = f"{path}/{DATAMODULE_CONFIG_NAME}"
         if os.path.isfile(datamodule_config):
             datamodule_configs.append(datamodule_config)
         else:
             datamodule_configs.append(None)
 
+        # Check for experiment config
+        # For backwards compatibility - this might always exist
+        experiment_config = f"{path}/{FULL_CONFIG_NAME}"
+        if os.path.isfile(datamodule_config):
+            experiment_configs.append(experiment_config)
+        else:
+            experiment_configs.append(None)
+
     if is_ensemble:
         model_config = {
             "_target_": "pvnet.models.ensemble.Ensemble",
@@ -90,7 +100,11 @@ def get_model_from_checkpoints(
         model_config = model_configs[0]
         model = models[0]
 
+    # Assume if using an ensemble that the members were trained on the same input data
     data_config = data_configs[0]
     datamodule_config = datamodule_configs[0]
 
-    return model, model_config, data_config, datamodule_config
+    # TODO: How should we save the experimental configs if we had an ensemble?
+    experiment_config = experiment_configs[0]
+
+    return model, model_config, data_config, datamodule_config, experiment_config

pvnet-5.0.0/pvnet/models/__init__.py

@@ -0,0 +1,4 @@
+"""Models for PVNet"""
+from .base_model import BaseModel
+from .ensemble import Ensemble
+from .late_fusion.late_fusion import LateFusionModel