PyPI - ocf-data-sampler - Versions diffs - 0.5.19__tar.gz → 0.5.22__tar.gz - Mend

ocf-data-sampler 0.5.19tar.gz → 0.5.22tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Potentially problematic release.

This version of ocf-data-sampler might be problematic. Click here for more details.

Files changed (72) hide show

{ocf_data_sampler-0.5.19 → ocf_data_sampler-0.5.22}/PKG-INFO RENAMED Viewed

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ocf-data-sampler
-Version: 0.5.19
+Version: 0.5.22
 Author: James Fulton, Peter Dudfield
 Author-email: Open Climate Fix team <info@openclimatefix.org>
 License: MIT License
@@ -50,7 +50,7 @@ Requires-Dist: zarr>=3
 # ocf-data-sampler
 <!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
-[![All Contributors](https://img.shields.io/badge/all_contributors-14-orange.svg?style=flat-square)](#contributors-)
+[![All Contributors](https://img.shields.io/badge/all_contributors-15-orange.svg?style=flat-square)](#contributors-)
 <!-- ALL-CONTRIBUTORS-BADGE:END -->
 [![tags badge](https://img.shields.io/github/v/tag/openclimatefix/ocf-data-sampler?include_prereleases&sort=semver&color=FFAC5F)](https://github.com/openclimatefix/ocf-data-sampler/tags)
@@ -137,6 +137,9 @@ Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/d
       <td align="center" valign="top" width="14.28%"><a href="https://drona-gyawali.github.io/"><img src="https://avatars.githubusercontent.com/u/170401554?v=4?s=100" width="100px;" alt="Dorna Raj Gyawali"/><br /><sub><b>Dorna Raj Gyawali</b></sub></a><br /><a href="https://github.com/openclimatefix/ocf-data-sampler/commits?author=drona-gyawali" title="Code">💻</a></td>
       <td align="center" valign="top" width="14.28%"><a href="https://github.com/adnanhashmi25"><img src="https://avatars.githubusercontent.com/u/55550094?v=4?s=100" width="100px;" alt="Adnan Hashmi"/><br /><sub><b>Adnan Hashmi</b></sub></a><br /><a href="https://github.com/openclimatefix/ocf-data-sampler/commits?author=adnanhashmi25" title="Code">💻</a></td>
     </tr>
+    <tr>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/utsav-pal"><img src="https://avatars.githubusercontent.com/u/159793156?v=4?s=100" width="100px;" alt="utsav-pal"/><br /><sub><b>utsav-pal</b></sub></a><br /><a href="https://github.com/openclimatefix/ocf-data-sampler/commits?author=utsav-pal" title="Code">💻</a></td>
+    </tr>
   </tbody>
 </table>

{ocf_data_sampler-0.5.19 → ocf_data_sampler-0.5.22}/README.md RENAMED Viewed

@@ -1,7 +1,7 @@
 # ocf-data-sampler
 <!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
-[![All Contributors](https://img.shields.io/badge/all_contributors-14-orange.svg?style=flat-square)](#contributors-)
+[![All Contributors](https://img.shields.io/badge/all_contributors-15-orange.svg?style=flat-square)](#contributors-)
 <!-- ALL-CONTRIBUTORS-BADGE:END -->
 [![tags badge](https://img.shields.io/github/v/tag/openclimatefix/ocf-data-sampler?include_prereleases&sort=semver&color=FFAC5F)](https://github.com/openclimatefix/ocf-data-sampler/tags)
@@ -88,6 +88,9 @@ Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/d
       <td align="center" valign="top" width="14.28%"><a href="https://drona-gyawali.github.io/"><img src="https://avatars.githubusercontent.com/u/170401554?v=4?s=100" width="100px;" alt="Dorna Raj Gyawali"/><br /><sub><b>Dorna Raj Gyawali</b></sub></a><br /><a href="https://github.com/openclimatefix/ocf-data-sampler/commits?author=drona-gyawali" title="Code">💻</a></td>
       <td align="center" valign="top" width="14.28%"><a href="https://github.com/adnanhashmi25"><img src="https://avatars.githubusercontent.com/u/55550094?v=4?s=100" width="100px;" alt="Adnan Hashmi"/><br /><sub><b>Adnan Hashmi</b></sub></a><br /><a href="https://github.com/openclimatefix/ocf-data-sampler/commits?author=adnanhashmi25" title="Code">💻</a></td>
     </tr>
+    <tr>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/utsav-pal"><img src="https://avatars.githubusercontent.com/u/159793156?v=4?s=100" width="100px;" alt="utsav-pal"/><br /><sub><b>utsav-pal</b></sub></a><br /><a href="https://github.com/openclimatefix/ocf-data-sampler/commits?author=utsav-pal" title="Code">💻</a></td>
+    </tr>
   </tbody>
 </table>

ocf_data_sampler-0.5.22/ocf_data_sampler/torch_datasets/datasets/picklecache.py ADDED Viewed

@@ -0,0 +1,33 @@
+"""A mixin to handle pickling and caching a dataset's state."""
+import os
+import pickle
+class PickleCacheMixin:
+    """A mixin for classes that need to cache their state using pickle."""
+    def __init__(self, *args: list, **kwargs: dict) -> None:
+        """Initialize the pickle path and call the parent constructor."""
+        self._pickle_path = None
+        super().__init__(*args, **kwargs)  # cooperative multiple inheritance
+    def presave_pickle(self, pickle_path: str) -> None:
+        """Save the full object state to a pickle file and store the pickle path."""
+        self._pickle_path = pickle_path
+        with open(pickle_path, "wb") as f:
+            pickle.dump(self.__dict__, f)
+    def __getstate__(self) -> dict:
+        """If presaved, only pickle reference. Otherwise pickle everything."""
+        if self._pickle_path:
+            return {"_pickle_path": self._pickle_path}
+        else:
+            return self.__dict__
+    def __setstate__(self, state: dict) -> None:
+        """Restore object from pickle, reloading from presaved file if possible."""
+        self.__dict__.update(state)
+        if self._pickle_path and os.path.exists(self._pickle_path):
+            with open(self._pickle_path, "rb") as f:
+                saved_state = pickle.load(f) # noqa: S301
+                self.__dict__.update(saved_state)

{ocf_data_sampler-0.5.19 → ocf_data_sampler-0.5.22}/ocf_data_sampler/torch_datasets/datasets/pvnet_uk.py RENAMED Viewed

@@ -19,6 +19,7 @@ from ocf_data_sampler.numpy_sample.common_types import NumpyBatch, NumpySample
 from ocf_data_sampler.numpy_sample.gsp import GSPSampleKey
 from ocf_data_sampler.numpy_sample.nwp import NWPSampleKey
 from ocf_data_sampler.select import Location, fill_time_periods
+from ocf_data_sampler.torch_datasets.datasets.picklecache import PickleCacheMixin
 from ocf_data_sampler.torch_datasets.utils import (
     add_alterate_coordinate_projections,
     config_normalization_values_to_dicts,
@@ -67,7 +68,7 @@ def get_gsp_locations(
     return locations
-class AbstractPVNetUKDataset(Dataset):
+class AbstractPVNetUKDataset(PickleCacheMixin, Dataset):
     """Abstract class for PVNet UK datasets."""
     def __init__(
@@ -85,6 +86,8 @@ class AbstractPVNetUKDataset(Dataset):
             end_time: Limit the init-times to be before this
             gsp_ids: List of GSP IDs to create samples for. Defaults to all
         """
+        super().__init__()
         config = load_yaml_configuration(config_filename)
         datasets_dict = get_dataset_dict(config.input_data, gsp_ids=gsp_ids)
@@ -225,7 +228,6 @@ class AbstractPVNetUKDataset(Dataset):
         return valid_t0_times
 class PVNetUKRegionalDataset(AbstractPVNetUKDataset):
     """A torch Dataset for creating PVNet UK regional samples."""

{ocf_data_sampler-0.5.19 → ocf_data_sampler-0.5.22}/ocf_data_sampler/torch_datasets/datasets/site.py RENAMED Viewed

@@ -24,6 +24,7 @@ from ocf_data_sampler.select import (
     find_contiguous_t0_periods,
     intersection_of_multiple_dataframes_of_periods,
 )
+from ocf_data_sampler.torch_datasets.datasets.picklecache import PickleCacheMixin
 from ocf_data_sampler.torch_datasets.utils import (
     add_alterate_coordinate_projections,
     config_normalization_values_to_dicts,
@@ -144,7 +145,7 @@ def process_and_combine_datasets(
     return combined_sample
-class SitesDataset(Dataset):
+class SitesDataset(PickleCacheMixin, Dataset):
     """A torch Dataset for creating PVNet Site samples."""
     def __init__(
@@ -160,6 +161,8 @@ class SitesDataset(Dataset):
             start_time: Limit the init-times to be after this
             end_time: Limit the init-times to be before this
         """
+        super().__init__()
         config = load_yaml_configuration(config_filename)
         datasets_dict = get_dataset_dict(config.input_data)
@@ -258,6 +261,9 @@ class SitesDataset(Dataset):
     @override
     def __getitem__(self, idx: int) -> dict:
+        idx = int(idx)
         # Get the coordinates of the sample
         t0, site_id = self.valid_t0_and_site_ids.iloc[idx]
@@ -301,7 +307,7 @@ class SitesDataset(Dataset):
         return self._get_sample(t0, location)
-class SitesDatasetConcurrent(Dataset):
+class SitesDatasetConcurrent(PickleCacheMixin, Dataset):
     """A torch Dataset for creating PVNet Site batches with samples for all sites."""
     def __init__(
@@ -317,6 +323,8 @@ class SitesDatasetConcurrent(Dataset):
             start_time: Limit the init-times to be after this
             end_time: Limit the init-times to be before this
         """
+        super().__init__()
         config = load_yaml_configuration(config_filename)
         datasets_dict = get_dataset_dict(config.input_data)

{ocf_data_sampler-0.5.19 → ocf_data_sampler-0.5.22}/ocf_data_sampler.egg-info/PKG-INFO RENAMED Viewed

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ocf-data-sampler
-Version: 0.5.19
+Version: 0.5.22
 Author: James Fulton, Peter Dudfield
 Author-email: Open Climate Fix team <info@openclimatefix.org>
 License: MIT License
@@ -50,7 +50,7 @@ Requires-Dist: zarr>=3
 # ocf-data-sampler
 <!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
-[![All Contributors](https://img.shields.io/badge/all_contributors-14-orange.svg?style=flat-square)](#contributors-)
+[![All Contributors](https://img.shields.io/badge/all_contributors-15-orange.svg?style=flat-square)](#contributors-)
 <!-- ALL-CONTRIBUTORS-BADGE:END -->
 [![tags badge](https://img.shields.io/github/v/tag/openclimatefix/ocf-data-sampler?include_prereleases&sort=semver&color=FFAC5F)](https://github.com/openclimatefix/ocf-data-sampler/tags)
@@ -137,6 +137,9 @@ Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/d
       <td align="center" valign="top" width="14.28%"><a href="https://drona-gyawali.github.io/"><img src="https://avatars.githubusercontent.com/u/170401554?v=4?s=100" width="100px;" alt="Dorna Raj Gyawali"/><br /><sub><b>Dorna Raj Gyawali</b></sub></a><br /><a href="https://github.com/openclimatefix/ocf-data-sampler/commits?author=drona-gyawali" title="Code">💻</a></td>
       <td align="center" valign="top" width="14.28%"><a href="https://github.com/adnanhashmi25"><img src="https://avatars.githubusercontent.com/u/55550094?v=4?s=100" width="100px;" alt="Adnan Hashmi"/><br /><sub><b>Adnan Hashmi</b></sub></a><br /><a href="https://github.com/openclimatefix/ocf-data-sampler/commits?author=adnanhashmi25" title="Code">💻</a></td>
     </tr>
+    <tr>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/utsav-pal"><img src="https://avatars.githubusercontent.com/u/159793156?v=4?s=100" width="100px;" alt="utsav-pal"/><br /><sub><b>utsav-pal</b></sub></a><br /><a href="https://github.com/openclimatefix/ocf-data-sampler/commits?author=utsav-pal" title="Code">💻</a></td>
+    </tr>
   </tbody>
 </table>

{ocf_data_sampler-0.5.19 → ocf_data_sampler-0.5.22}/ocf_data_sampler.egg-info/SOURCES.txt RENAMED Viewed

@@ -49,6 +49,7 @@ ocf_data_sampler/select/location.py
 ocf_data_sampler/select/select_spatial_slice.py
 ocf_data_sampler/select/select_time_slice.py
 ocf_data_sampler/torch_datasets/datasets/__init__.py
+ocf_data_sampler/torch_datasets/datasets/picklecache.py
 ocf_data_sampler/torch_datasets/datasets/pvnet_uk.py
 ocf_data_sampler/torch_datasets/datasets/site.py
 ocf_data_sampler/torch_datasets/sample/__init__.py