ocf-data-sampler 0.0.19__py3-none-any.whl → 0.0.43__py3-none-any.whl

ocf_data_sampler-0.0.43.dist-info/METADATA

@@ -0,0 +1,154 @@
+Metadata-Version: 2.2
+Name: ocf_data_sampler
+Version: 0.0.43
+Summary: Sample from weather data for renewable energy prediction
+Author: James Fulton, Peter Dudfield, and the Open Climate Fix team
+Author-email: info@openclimatefix.org
+Maintainer: Open Climate Fix Ltd
+License: MIT License
+        
+        Copyright (c) 2023 Open Climate Fix
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: homepage, https://github.com/openclimatefix
+Project-URL: repository, https://github.com/openclimatefix/ocf-data-sampler
+Keywords: weather data,renewable energy prediction,sample weather data
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Operating System :: POSIX :: Linux
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch
+Requires-Dist: numpy
+Requires-Dist: pandas
+Requires-Dist: xarray
+Requires-Dist: zarr==2.18.3
+Requires-Dist: dask
+Requires-Dist: ocf_blosc2
+Requires-Dist: pvlib
+Requires-Dist: pydantic
+Requires-Dist: pyproj
+Requires-Dist: pathy
+Requires-Dist: pyaml_env
+Requires-Dist: pyresample
+Provides-Extra: docs
+Requires-Dist: mkdocs>=1.2; extra == "docs"
+Requires-Dist: mkdocs-material>=8.0; extra == "docs"
+
+# ocf-data-sampler
+ 
+<!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
+[![All Contributors](https://img.shields.io/badge/all_contributors-10-orange.svg?style=flat-square)](#contributors-)
+<!-- ALL-CONTRIBUTORS-BADGE:END -->
+
+[![tags badge](https://img.shields.io/github/v/tag/openclimatefix/ocf-data-sampler?include_prereleases&sort=semver&color=FFAC5F)](https://github.com/openclimatefix/ocf-data-sampler/tags)
+[![ease of contribution: easy](https://img.shields.io/badge/ease%20of%20contribution:%20easy-32bd50)](https://github.com/openclimatefix#how-easy-is-it-to-get-involved) 
+
+**ocf-data-sampler** contains all the tools needed to create samples and feed them to our models, such as [PVNet](https://github.com/openclimatefix/PVNet/). The data we work with—typically energy data, satellite imagery, and numerical weather predictions (NWPs)—is usually too heavy to do this on the fly, so that's where this repo comes in: handling steps like opening the data, selecting the right samples, normalising and reshaping, and saving to and reading from disk.
+
+We are currently migrating to this repo from [ocf_datapipes](https://github.com/openclimatefix/ocf_datapipes/), which performs the same functions but is built around `PyTorch DataPipes`, which are quite cumbersome to work with and are no longer maintained by PyTorch. **ocf-data-sampler** uses `PyTorch Datasets`, and we've taken the opportunity to make the code much cleaner and more manageable.
+
+> [!Note]
+> This repository is still in development and does not yet have the full 
+> functionality of its predecessor, [ocf_datapipes](https://github.com/openclimatefix/ocf_datapipes/).
+> It might not be ready for use out of the box! We would really appreciate any help to let us make the transition faster.
+
+## Documentation
+
+**ocf-data-sampler** doesn't have external documentation _yet_; you can read a bit about how our torch datasets work in the README [here](ocf_data_sampler/torch_datasets/README.md).
+
+
+## FAQ
+
+If you have any questions about this or any other of our repos, don't hesitate to hop to our [Discussions Page](https://github.com/orgs/openclimatefix/discussions)!
+
+### How does ocf-data-sampler deal with data sources that use different projections (e.g. some are in latitude-longitude, and some in OSGB)?
+
+When creating samples, we make a spatial crop of a preset size centred around a point of interest (POI, usually a solar or wind farm). The size of the crop is set not in miles or kilometres, but in 'pixels', which would be different for different data sources, depending on their spatial resolution, projections they use, and where the POI is. For example, a latitude-longitude source with a 1° resolution will have pixel sizes corresponding to very different 'surface' distances (that you might measure in, e.g., kilometres) from a source with 0.1° resolution. The pixel size will even be different for the same source depending on how close the POI is to the equator!
+
+Instead of trying to accommodate for all these differences and make all the sources use the same spatial grid, we translate the POI's position into the corresponding coordinate system and select the crop using the source's original grid. This 'snapshot' is then passed to the model with no additional information on what specific coordinates it represents; instead, since the size is always the same and the POI is always in the centre, the model gets consistent information on the measurements at a location near the POI and how it affects the target, without any explicit knowledge of where that location is in coordinate system terms.
+
+## Development
+
+You can install **ocf-data-sampler** for development as follows:
+
+``` 
+pip install git+https://github.com/openclimatefix/ocf-data-sampler.git
+```
+
+### Running the test suite
+
+The tests in this project use `pytest`. Once you have it installed, you can run it from the project's directory:
+
+```
+cd ocf-data-sampler
+pytest
+``` 
+
+## Contributing and community
+
+[![issues badge](https://img.shields.io/github/issues/openclimatefix/ocf-data-sampler?color=FFAC5F)](https://github.com/openclimatefix/ocf-data-sampler/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc)
+
+- PR's are welcome! See the [Organisation Profile](https://github.com/openclimatefix) for details on contributing
+- Find out about our other projects in the [OCF Meta Repo](https://github.com/openclimatefix/ocf-meta-repo)
+- Check out the [OCF blog](https://openclimatefix.org/blog) for updates
+- Follow OCF on [LinkedIn](https://uk.linkedin.com/company/open-climate-fix)
+
+
+## Contributors
+
+Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):
+
+<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
+<!-- prettier-ignore-start -->
+<!-- markdownlint-disable -->
+<table>
+  <tbody>
+    <tr>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/dfulu"><img src="https://avatars.githubusercontent.com/u/41546094?v=4?s=100" width="100px;" alt="James Fulton"/><br /><sub><b>James Fulton</b></sub></a><br /><a href="https://github.com/openclimatefix/ocf-data-sampler/commits?author=dfulu" title="Code">💻</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/AUdaltsova"><img src="https://avatars.githubusercontent.com/u/43303448?v=4?s=100" width="100px;" alt="Alexandra Udaltsova"/><br /><sub><b>Alexandra Udaltsova</b></sub></a><br /><a href="https://github.com/openclimatefix/ocf-data-sampler/commits?author=AUdaltsova" title="Code">💻</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/Sukh-P"><img src="https://avatars.githubusercontent.com/u/42407101?v=4?s=100" width="100px;" alt="Sukhil Patel"/><br /><sub><b>Sukhil Patel</b></sub></a><br /><a href="https://github.com/openclimatefix/ocf-data-sampler/commits?author=Sukh-P" title="Code">💻</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/peterdudfield"><img src="https://avatars.githubusercontent.com/u/34686298?v=4?s=100" width="100px;" alt="Peter Dudfield"/><br /><sub><b>Peter Dudfield</b></sub></a><br /><a href="https://github.com/openclimatefix/ocf-data-sampler/commits?author=peterdudfield" title="Code">💻</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/VikramsDataScience"><img src="https://avatars.githubusercontent.com/u/45002417?v=4?s=100" width="100px;" alt="Vikram Pande"/><br /><sub><b>Vikram Pande</b></sub></a><br /><a href="https://github.com/openclimatefix/ocf-data-sampler/commits?author=VikramsDataScience" title="Code">💻</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/SophiaLi20"><img src="https://avatars.githubusercontent.com/u/163532536?v=4?s=100" width="100px;" alt="Unnati Bhardwaj"/><br /><sub><b>Unnati Bhardwaj</b></sub></a><br /><a href="https://github.com/openclimatefix/ocf-data-sampler/commits?author=SophiaLi20" title="Documentation">📖</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/alirashidAR"><img src="https://avatars.githubusercontent.com/u/110668489?v=4?s=100" width="100px;" alt="Ali Rashid"/><br /><sub><b>Ali Rashid</b></sub></a><br /><a href="https://github.com/openclimatefix/ocf-data-sampler/commits?author=alirashidAR" title="Code">💻</a></td>
+    </tr>
+    <tr>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/felix-e-h-p"><img src="https://avatars.githubusercontent.com/u/137530077?v=4?s=100" width="100px;" alt="Felix"/><br /><sub><b>Felix</b></sub></a><br /><a href="https://github.com/openclimatefix/ocf-data-sampler/commits?author=felix-e-h-p" title="Code">💻</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://timothyajaniportfolio-b6v3zq29k-timthegreat.vercel.app/"><img src="https://avatars.githubusercontent.com/u/60073728?v=4?s=100" width="100px;" alt="Ajani Timothy"/><br /><sub><b>Ajani Timothy</b></sub></a><br /><a href="https://github.com/openclimatefix/ocf-data-sampler/commits?author=Tim1119" title="Code">💻</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://rupeshmangalam.vercel.app/"><img src="https://avatars.githubusercontent.com/u/91172425?v=4?s=100" width="100px;" alt="Rupesh Mangalam"/><br /><sub><b>Rupesh Mangalam</b></sub></a><br /><a href="https://github.com/openclimatefix/ocf-data-sampler/commits?author=RupeshMangalam21" title="Code">💻</a></td>
+    </tr>
+  </tbody>
+</table>
+
+<!-- markdownlint-restore -->
+<!-- prettier-ignore-end -->
+
+<!-- ALL-CONTRIBUTORS-LIST:END -->
+
+This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!
+
+---
+
+*Part of the [Open Climate Fix](https://github.com/orgs/openclimatefix/people) community.*
+
+[![OCF Logo](https://cdn.prod.website-files.com/62d92550f6774db58d441cca/6324a2038936ecda71599a8b_OCF_Logo_black_trans.png)](https://openclimatefix.org)
+

ocf_data_sampler-0.0.43.dist-info/RECORD

@@ -0,0 +1,71 @@
+ocf_data_sampler/__init__.py,sha256=AbpHGcgLb-kRsJGnwFEktk7uzpZOCcBY74-YBdrKVGs,1
+ocf_data_sampler/constants.py,sha256=G2VfkE_-veq_0hNBQQOQCtCsfC37O5-QG9mJWEmln5s,4153
+ocf_data_sampler/utils.py,sha256=rKA0BHAyAG4f90zEcgxp25EEYrXS-aOVNzttZ6Mzv2k,250
+ocf_data_sampler/config/__init__.py,sha256=YXnAkgHViHB26hSsjiv32b6EbpG-A1kKTkARJf0_RkY,212
+ocf_data_sampler/config/load.py,sha256=4f7vPHAIAmd-55tPxoIzn7F_TI_ue4NxkDcLPoVWl0g,943
+ocf_data_sampler/config/model.py,sha256=sXmh7IadwXDT-7lxEl5_b3vjovZgZYR77EXy4GHaf4w,7276
+ocf_data_sampler/config/save.py,sha256=JhPyX1DJAW-mi3GfkDF4nlrEXNt-yfX9ibBy8CZ6ENI,2464
+ocf_data_sampler/data/uk_gsp_locations.csv,sha256=RSh7DRh55E3n8lVAaWXGTaXXHevZZtI58td4d4DhGos,10415772
+ocf_data_sampler/load/__init__.py,sha256=MjgfxilTzyz1RYFoBEeAXmE9hyjknLvdmlHPmlAoiQY,44
+ocf_data_sampler/load/gsp.py,sha256=Gcr1JVUOPKhFRDCSHtfPDjxx0BtyyEhXrZvGEKLPJ5I,759
+ocf_data_sampler/load/load_dataset.py,sha256=Ua3RaUg4PIYJkD9BKqTfN8IWUbezbhThJGgEkd9PcaE,1587
+ocf_data_sampler/load/satellite.py,sha256=3KlA1fx4SwxdzM-jC1WRaONXO0D6m0WxORnEnwUnZrA,2967
+ocf_data_sampler/load/site.py,sha256=P83uz01WBDzoZajdOH0m8FQt4-buKDlUk19N548KqhA,1086
+ocf_data_sampler/load/utils.py,sha256=EQGvVWlGMoSOdbDYuMfVAa0v6wmAOPmHIAemdrTB5v4,1406
+ocf_data_sampler/load/nwp/__init__.py,sha256=SmcrnbygO5xtCKmGR4wtHrj-HI7nOAvnAtfuvRufBGQ,25
+ocf_data_sampler/load/nwp/nwp.py,sha256=O4QnajEZem8BvBgTcYYDBhRhgqPYuJkolHmpMRmrXEA,610
+ocf_data_sampler/load/nwp/providers/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+ocf_data_sampler/load/nwp/providers/ecmwf.py,sha256=2iR1Iy542lo51rC6XFLV-3pbUE68dWjlHa6TVJzx3ac,1280
+ocf_data_sampler/load/nwp/providers/ukv.py,sha256=79Bm7q-K_GJPYMy62SUIZbRWRF4-tIaB1dYPEgLD9vo,1207
+ocf_data_sampler/load/nwp/providers/utils.py,sha256=Sy2exG1wpXLLhMXYdsfR-DZMR3txG1_bBmBdchlc-yA,848
+ocf_data_sampler/numpy_sample/__init__.py,sha256=jWvN4BDYAgoFkdGBxjNrYioZdZNVbIcP4RsNGJCEGm0,345
+ocf_data_sampler/numpy_sample/collate.py,sha256=y8QFUhskaAfOMP22aVkexwyGAwLHbNE-q1pOZ6txWKA,2227
+ocf_data_sampler/numpy_sample/gsp.py,sha256=4ZckEN7S82UY36e9z6rp5DTp10SAn_cNISBNilpOADw,1008
+ocf_data_sampler/numpy_sample/nwp.py,sha256=eEIRXP09wY99Vq-ItnfQ6U9AY1vJPLYKfUOc83V6ICo,1302
+ocf_data_sampler/numpy_sample/satellite.py,sha256=WYG9_ryOk3RWHWjgcMjIaU-YahEhTXz2xN0rQm-_TTQ,914
+ocf_data_sampler/numpy_sample/site.py,sha256=E2MQKkgUXikbIlKYU4ueNCPoBYJC-vnkbrQ_IwP2hqw,792
+ocf_data_sampler/numpy_sample/sun_position.py,sha256=UklhucCxCT6GMlAhCWL6c4cfWrdc1cWgegrYaqUoHOY,1611
+ocf_data_sampler/select/__init__.py,sha256=E4AJulEbO2K-o0UlG1fgaEteuf_1ZFjHTvrotXSb4YU,332
+ocf_data_sampler/select/dropout.py,sha256=HCx5Wzk8Oh2Z9vV94Jy-ALJsHtGduwvMaQOleQXp5z0,1142
+ocf_data_sampler/select/fill_time_periods.py,sha256=iTtMjIPFYG5xtUYYedAFBLjTWWUa7t7WQ0-yksWf0-E,440
+ocf_data_sampler/select/find_contiguous_time_periods.py,sha256=q7IaNfX95A3z9XHqbhgtkZ4Js1gn5K9Qyp6DVLbsL-Q,11093
+ocf_data_sampler/select/geospatial.py,sha256=4xL-9y674jjoaXeqE52NHCHVfknciE4OEGsZtn9DvP4,4911
+ocf_data_sampler/select/location.py,sha256=26Y5ZjfFngShBwXieuWSoOA-RLaRzci4TTmcDk3Wg7U,2015
+ocf_data_sampler/select/select_spatial_slice.py,sha256=WNxwur9Q5oetvogATw8-hNejDuEwrXHzuZIovFDjNJA,11488
+ocf_data_sampler/select/select_time_slice.py,sha256=D5P_cSvnv8Qs49K5au7lPxDr9U_VmDn42s5leMzHt0k,6122
+ocf_data_sampler/select/spatial_slice_for_dataset.py,sha256=3tRrMBXr7s4CnClbVSIq7hpls3H4Y3qYTDwswcxCCCE,1763
+ocf_data_sampler/select/time_slice_for_dataset.py,sha256=LMw8KnOCKnPjD0m4UubAWERpaiQtzRKkI2cSh5a0A-M,4335
+ocf_data_sampler/torch_datasets/__init__.py,sha256=nJUa2KzVa84ZoM0PT2AbDz26ennmAYc7M7WJVfypPMs,85
+ocf_data_sampler/torch_datasets/process_and_combine.py,sha256=ygRBf70Jd6turmwLrv0Fsxo_Ge-54N5S2NUntyRxtJk,4305
+ocf_data_sampler/torch_datasets/pvnet_uk_regional.py,sha256=QRFqbdfNchVWj4y70n-rJdFvFGvQj-WpZLdFqWjnOTw,5543
+ocf_data_sampler/torch_datasets/site.py,sha256=2fN2tw5tCXoOkh8vzRWWm_5FNGglGwJInTanJgV4Xuo,14455
+ocf_data_sampler/torch_datasets/valid_time_periods.py,sha256=Qo65qUHtle_bW5tLTYr7empHTRv-lpjvfx_6GNJj3Xg,4371
+scripts/refactor_site.py,sha256=asZ27hQ4IyXgCCUaFJqcz1ObBNcV2W3ywqHBpSXA_fc,1728
+tests/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+tests/conftest.py,sha256=68hH-HPdHPLvLrtYJU8bjfkdGKbhPfNveLKvUs6_Lr0,7970
+tests/config/test_config.py,sha256=eaye_F7-el4tTP4n2vRME8qlV0b2jaKUX4HhgOUpa7E,5203
+tests/config/test_save.py,sha256=rA_XVxP1pOxB--5Ebujz4T5o-VbcrCbg2VSlSq2iI0o,1318
+tests/load/test_load_gsp.py,sha256=aT_nqaSXmUTcdHzuTT7AmXJr3R31k4OEN-Fv3eLxlQE,424
+tests/load/test_load_nwp.py,sha256=3qyyDkB1q9t3tyAwogfotNrxqUOpXXimco1CImoEWGg,753
+tests/load/test_load_satellite.py,sha256=STX5AqqmOAgUgE9R1xyq_sM3P1b8NKdGjO-hDhayfxM,524
+tests/load/test_load_sites.py,sha256=T9lSEnGPI8FQISudVYHHNTHeplNS62Vrx48jaZ6J_Jo,364
+tests/numpy_sample/test_collate.py,sha256=oWVNRR0LyF0iqaxjsq6hPMmkSGIyvGVlAM6q2rqvXqQ,840
+tests/numpy_sample/test_gsp.py,sha256=FLlq4SlJ-9cSRAepf4_ksA6PsUVKegnKEAc5pUojCJ0,1458
+tests/numpy_sample/test_nwp.py,sha256=yf4u7mAU0E3FQ4xAH6YjuHuHBzzFoXjHSFNkOVJUdSM,1455
+tests/numpy_sample/test_satellite.py,sha256=cCqtn5See-uSNfh89COGTUQNuFm6sIZ8QmBVHsuUeRI,1189
+tests/numpy_sample/test_sun_position.py,sha256=_ENYzsNBVPdNXf--FI-UUFqw2u5w7_zqw6LcENU2uZM,2504
+tests/select/test_dropout.py,sha256=kiycl7RxAQYMCZJlokmx6Da5h_oBpSs8Is8pmSW4gOU,2413
+tests/select/test_fill_time_periods.py,sha256=o59f2YRe5b0vJrG3B0aYZkYeHnpNk4s6EJxdXZluNQg,907
+tests/select/test_find_contiguous_time_periods.py,sha256=kOga_V7er5We7ewMARXaKdM3agOhsvZYx8inXtUn1PM,5976
+tests/select/test_location.py,sha256=_WZk2FPYeJ-nIfCJS6Sp_yaVEEo7m31DmMFoZzgyCts,2712
+tests/select/test_select_spatial_slice.py,sha256=7EX9b6g-pMdACQx3yefjs5do2s-Rho2UmKevV4oglsU,5147
+tests/select/test_select_time_slice.py,sha256=K1EJR5TwZa9dJf_YTEHxGtvs398iy1xS2lr1BgJZkoo,9603
+tests/torch_datasets/conftest.py,sha256=eRCzHE7cxS4AoskExkCGFDBeqItktAYNAdkfpMoFCeE,629
+tests/torch_datasets/test_process_and_combine.py,sha256=mbjQdqzLhox-U2sc1Ec68xLz95b3XOyPa7WchgxUM88,4256
+tests/torch_datasets/test_pvnet_uk_regional.py,sha256=dLY861PMyQ_buTksP8d0UXzfKsZ_CFNgceSYVGXRIRs,2134
+tests/torch_datasets/test_site.py,sha256=TSmdpD0OFid6e8eRgeB5FGxglUN1L2t4cIuMTfjA7jE,4539
+ocf_data_sampler-0.0.43.dist-info/LICENSE,sha256=F-Q3UFCR-BECSocV55BFDpn4YKxve9PKrm-lTt6o_Tg,1073
+ocf_data_sampler-0.0.43.dist-info/METADATA,sha256=pqYvFJO9iGZnsMneHiF0WobuYrSpfxG_Cw701uBi480,11431
+ocf_data_sampler-0.0.43.dist-info/WHEEL,sha256=In9FTNxeP60KnTkGw7wk6mJPYd_dQSjEZmXdBdMCI-8,91
+ocf_data_sampler-0.0.43.dist-info/top_level.txt,sha256=Faob6N6cFdPc5eUpCTYcXgCaNhi4XLLteUL5W5ayYmg,31
+ocf_data_sampler-0.0.43.dist-info/RECORD,,

{ocf_data_sampler-0.0.19.dist-info → ocf_data_sampler-0.0.43.dist-info}/WHEEL

@@ -1,5 +1,5 @@
 Wheel-Version: 1.0
-Generator: setuptools (75.1.0)
+Generator: setuptools (75.8.0)
 Root-Is-Purelib: true
 Tag: py3-none-any
 

{ocf_data_sampler-0.0.19.dist-info → ocf_data_sampler-0.0.43.dist-info}/top_level.txt

@@ -1,2 +1,3 @@
 ocf_data_sampler
+scripts
 tests

scripts/refactor_site.py

@@ -0,0 +1,50 @@
+""" Helper functions for refactoring legacy site data """
+
+
+def legacy_format(data_ds, metadata_df):
+    """This formats old legacy data to the new format.
+
+    1. This renames the columns in the metadata
+    2. Re-formats the site data from data variables named by the site_id to
+    a data array with a site_id dimension. Also adds capacity_kwp to the dataset as a time series for each site_id
+    """
+
+    if "system_id" in metadata_df.columns:
+        metadata_df["site_id"] = metadata_df["system_id"]
+
+    if "capacity_megawatts" in metadata_df.columns:
+        metadata_df["capacity_kwp"] = metadata_df["capacity_megawatts"] * 1000
+
+    # only site data has the site_id as data variables.
+    # We want to join them all together and create another coordinate called site_id
+    if "0" in data_ds:
+        gen_df = data_ds.to_dataframe()
+        gen_da = xr.DataArray(
+            data=gen_df.values,
+            coords=(
+                ("time_utc", gen_df.index.values),
+                ("site_id", metadata_df["site_id"]),
+            ),
+            name="generation_kw",
+        )
+
+        capacity_df = gen_df
+        for col in capacity_df.columns:
+            capacity_df[col] = metadata_df[metadata_df["site_id"].astype(str) == col][
+                "capacity_kwp"
+            ].iloc[0]
+        capacity_da = xr.DataArray(
+            data=capacity_df.values,
+            coords=(
+                ("time_utc", gen_df.index.values),
+                ("site_id", metadata_df["site_id"]),
+            ),
+            name="capacity_kwp",
+        )
+        data_ds = xr.Dataset(
+            {
+                "generation_kw": gen_da,
+                "capacity_kwp": capacity_da,
+            }
+        )
+    return data_ds

tests/config/test_config.py

@@ -0,0 +1,161 @@
+import tempfile
+
+import pytest
+from pydantic import ValidationError
+
+from ocf_data_sampler.config import (
+    load_yaml_configuration,
+    Configuration,
+    save_yaml_configuration
+)
+
+
+def test_default_configuration():
+    """Test default pydantic class"""
+
+    _ = Configuration()
+
+
+def test_load_yaml_configuration(test_config_filename):
+    """
+    Test that yaml loading works for 'test_config.yaml'
+    and fails for an empty .yaml file
+    """
+
+    # check we get an error if loading a file with no config
+    with tempfile.NamedTemporaryFile(suffix=".yaml") as fp:
+        filename = fp.name
+
+        # check that temp file can't be loaded
+        with pytest.raises(TypeError):
+            _ = load_yaml_configuration(filename)
+
+    # test can load test_config.yaml
+    config = load_yaml_configuration(test_config_filename)
+
+    assert isinstance(config, Configuration)
+
+
+def test_yaml_save(test_config_filename):
+    """
+    Check configuration can be saved to a .yaml file
+    """
+
+    test_config = load_yaml_configuration(test_config_filename)
+
+    with tempfile.NamedTemporaryFile(suffix=".yaml") as fp:
+        filename = fp.name
+
+        # save default config to file
+        save_yaml_configuration(test_config, filename)
+
+        # check the file can be loaded back
+        tmp_config = load_yaml_configuration(filename)
+
+        # check loaded configuration is the same as the one passed to save
+        assert test_config == tmp_config
+
+
+def test_extra_field_error():
+    """
+    Check an extra parameters in config causes error
+    """
+
+    configuration = Configuration()
+    configuration_dict = configuration.model_dump()
+    configuration_dict["extra_field"] = "extra_value"
+    with pytest.raises(ValidationError, match="Extra inputs are not permitted"):
+        _ = Configuration(**configuration_dict)
+
+
+def test_incorrect_interval_start_minutes(test_config_filename):
+    """
+    Check a history length not divisible by time resolution causes error
+    """
+
+    configuration = load_yaml_configuration(test_config_filename)
+
+    configuration.input_data.nwp['ukv'].interval_start_minutes = -1111
+    with pytest.raises(
+        ValueError, 
+        match="interval_start_minutes must be divisible by time_resolution_minutes"
+    ):
+        _ = Configuration(**configuration.model_dump())
+
+
+def test_incorrect_interval_end_minutes(test_config_filename):
+    """
+    Check a forecast length not divisible by time resolution causes error
+    """
+
+    configuration = load_yaml_configuration(test_config_filename)
+
+    configuration.input_data.nwp['ukv'].interval_end_minutes = 1111
+    with pytest.raises(
+        ValueError, 
+        match="interval_end_minutes must be divisible by time_resolution_minutes"
+    ):
+        _ = Configuration(**configuration.model_dump())
+
+
+def test_incorrect_nwp_provider(test_config_filename):
+    """
+    Check an unexpected nwp provider causes error
+    """
+
+    configuration = load_yaml_configuration(test_config_filename)
+
+    configuration.input_data.nwp['ukv'].provider = "unexpected_provider"
+    with pytest.raises(Exception, match="NWP provider"):
+        _ = Configuration(**configuration.model_dump())
+
+
+def test_incorrect_dropout(test_config_filename):
+    """
+    Check a dropout timedelta over 0 causes error and 0 doesn't
+    """
+
+    configuration = load_yaml_configuration(test_config_filename)
+
+    # check a positive number is not allowed
+    configuration.input_data.nwp['ukv'].dropout_timedeltas_minutes = [120]
+    with pytest.raises(Exception, match="Dropout timedeltas must be negative"):
+        _ = Configuration(**configuration.model_dump())
+
+    # check 0 is allowed
+    configuration.input_data.nwp['ukv'].dropout_timedeltas_minutes = [0]
+    _ = Configuration(**configuration.model_dump())
+
+
+def test_incorrect_dropout_fraction(test_config_filename):
+    """
+    Check dropout fraction outside of range causes error
+    """
+
+    configuration = load_yaml_configuration(test_config_filename)
+
+    configuration.input_data.nwp['ukv'].dropout_fraction= 1.1
+
+    with pytest.raises(ValidationError,  match="Input should be less than or equal to 1"):
+        _ = Configuration(**configuration.model_dump())
+
+    configuration.input_data.nwp['ukv'].dropout_fraction= -0.1
+    with pytest.raises(ValidationError, match="Input should be greater than or equal to 0"):
+        _ = Configuration(**configuration.model_dump())
+
+
+def test_inconsistent_dropout_use(test_config_filename):
+    """
+    Check dropout fraction outside of range causes error
+    """
+
+    configuration = load_yaml_configuration(test_config_filename)
+    configuration.input_data.satellite.dropout_fraction= 1.0
+    configuration.input_data.satellite.dropout_timedeltas_minutes = None
+
+    with pytest.raises(ValueError, match="To dropout fraction > 0 requires a list of dropout timedeltas"):
+        _ = Configuration(**configuration.model_dump())
+    configuration.input_data.satellite.dropout_fraction= 0.0
+    configuration.input_data.satellite.dropout_timedeltas_minutes = [-120, -60]
+    with pytest.raises(ValueError, match="To use dropout timedeltas dropout fraction should be > 0"):
+        _ = Configuration(**configuration.model_dump())

tests/config/test_save.py

@@ -0,0 +1,37 @@
+"""Tests for configuration saving functionality."""
+import pytest
+from pathlib import Path
+import tempfile
+import yaml
+
+from ocf_data_sampler.config import Configuration, save_yaml_configuration
+
+@pytest.fixture
+def temp_dir():
+    """Create a temporary directory."""
+    with tempfile.TemporaryDirectory() as tmpdirname:
+        yield Path(tmpdirname)
+
+def test_save_yaml_configuration_basic(temp_dir):
+    """Test basic configuration saving functionality."""
+    config = Configuration()
+    filepath = temp_dir / "config.yaml"
+    result = save_yaml_configuration(config, filepath)
+    
+    assert filepath.exists()
+    with open(filepath) as f:
+        loaded_yaml = yaml.safe_load(f)
+    assert isinstance(loaded_yaml, dict)
+
+def test_save_yaml_configuration_none_filename():
+    """Test that None filename raises ValueError."""
+    config = Configuration()
+    with pytest.raises(ValueError, match="filename cannot be None"):
+        save_yaml_configuration(config, None)
+
+def test_save_yaml_configuration_invalid_directory(temp_dir):
+    """Test handling of invalid directory paths."""
+    config = Configuration()
+    invalid_path = (temp_dir / "nonexistent" / "config.yaml").resolve()
+    with pytest.raises(ValueError, match="Directory does not exist"):
+        save_yaml_configuration(config, invalid_path)

tests/conftest.py

@@ -6,11 +6,19 @@ import pytest
 import xarray as xr
 import tempfile
 
+from ocf_data_sampler.config.model import Site
+from ocf_data_sampler.config import load_yaml_configuration, save_yaml_configuration
+
+_top_test_directory = os.path.dirname(os.path.realpath(__file__))
+
+@pytest.fixture()
+def test_config_filename():
+    return f"{_top_test_directory}/test_data/configs/test_config.yaml"
 
 
 @pytest.fixture(scope="session")
 def config_filename():
-    return f"{os.path.dirname(os.path.abspath(__file__))}/test_data/pvnet_test_config.yaml"
+    return f"{os.path.dirname(os.path.abspath(__file__))}/test_data/configs/pvnet_test_config.yaml"
 
 
 @pytest.fixture(scope="session")
@@ -192,6 +200,68 @@ def ds_uk_gsp():
     })
 
 
+@pytest.fixture(scope="session")
+def data_sites() -> Site:
+    """
+    Make fake data for sites
+    Returns: filename for netcdf file, and csv metadata
+    """
+    times = pd.date_range("2023-01-01 00:00", "2023-01-02 00:00", freq="30min")
+    site_ids = list(range(0,10))
+    capacity_kwp_1d = np.array([0.1,1.1,4,6,8,9,15,2,3,4])
+    # these are quite specific for the fake satellite data
+    longitude = np.arange(-4, -3, 0.1)
+    latitude = np.arange(51, 52, 0.1)
+
+    generation = np.random.uniform(0, 200, size=(len(times), len(site_ids))).astype(np.float32)
+
+    # repeat capacity in new dims len(times) times
+    capacity_kwp = (np.tile(capacity_kwp_1d, len(times))).reshape(len(times),10)
+
+    coords = (
+        ("time_utc", times),
+        ("site_id", site_ids),
+    )
+
+    da_cap = xr.DataArray(
+        capacity_kwp,
+        coords=coords,
+    )
+
+    da_gen = xr.DataArray(
+        generation,
+        coords=coords,
+    )
+
+    # metadata
+    meta_df = pd.DataFrame(columns=[], data = [])
+    meta_df['site_id'] = site_ids
+    meta_df['capacity_kwp'] = capacity_kwp_1d
+    meta_df['longitude'] = longitude
+    meta_df['latitude'] = latitude
+
+    generation = xr.Dataset({
+        "capacity_kwp": da_cap,
+        "generation_kw": da_gen,
+    })
+
+    with tempfile.TemporaryDirectory() as tmpdir:
+        filename = tmpdir + "/sites.netcdf"
+        filename_csv = tmpdir + "/sites_metadata.csv"
+        generation.to_netcdf(filename)
+        meta_df.to_csv(filename_csv)
+
+        site = Site(
+            file_path=filename,
+            metadata_file_path=filename_csv,
+            interval_start_minutes=-30,
+            interval_end_minutes=60,
+            time_resolution_minutes=30,
+        )
+
+        yield site
+
+
 @pytest.fixture(scope="session")
 def uk_gsp_zarr_path(ds_uk_gsp):
 
@@ -200,3 +270,18 @@ def uk_gsp_zarr_path(ds_uk_gsp):
         ds_uk_gsp.to_zarr(filename)
         yield filename
 
+
+@pytest.fixture()
+def pvnet_config_filename(
+    tmp_path, config_filename, nwp_ukv_zarr_path, uk_gsp_zarr_path, sat_zarr_path
+):
+
+    # adjust config to point to the zarr file
+    config = load_yaml_configuration(config_filename)
+    config.input_data.nwp["ukv"].zarr_path = nwp_ukv_zarr_path
+    config.input_data.satellite.zarr_path = sat_zarr_path
+    config.input_data.gsp.zarr_path = uk_gsp_zarr_path
+
+    filename = f"{tmp_path}/configuration.yaml"
+    save_yaml_configuration(config, filename)
+    return filename

tests/load/test_load_gsp.py

@@ -0,0 +1,15 @@
+from ocf_data_sampler.load.gsp import open_gsp
+import xarray as xr
+
+
+def test_open_gsp(uk_gsp_zarr_path):
+    da = open_gsp(uk_gsp_zarr_path)
+
+    assert isinstance(da, xr.DataArray)
+    assert da.dims == ("time_utc", "gsp_id")
+
+    assert "nominal_capacity_mwp" in da.coords
+    assert "effective_capacity_mwp" in da.coords
+    assert "x_osgb" in da.coords
+    assert "y_osgb" in da.coords
+    assert da.shape == (49, 318)

tests/load/test_load_nwp.py

@@ -0,0 +1,21 @@
+import pandas as pd
+from xarray import DataArray
+import numpy as np
+
+from ocf_data_sampler.load.nwp import open_nwp
+
+
+def test_load_ukv(nwp_ukv_zarr_path):
+    da = open_nwp(zarr_path=nwp_ukv_zarr_path, provider="ukv")
+    assert isinstance(da, DataArray)
+    assert da.dims == ("init_time_utc", "step", "channel", "x_osgb", "y_osgb")
+    assert da.shape == (24 * 7, 11, 4, 50, 100)
+    assert np.issubdtype(da.dtype, np.number)
+
+
+def test_load_ecmwf(nwp_ecmwf_zarr_path):
+    da = open_nwp(zarr_path=nwp_ecmwf_zarr_path, provider="ecmwf")
+    assert isinstance(da, DataArray)
+    assert da.dims == ("init_time_utc", "step", "channel", "longitude", "latitude")
+    assert da.shape == (24 * 7, 15, 3, 15, 12)
+    assert np.issubdtype(da.dtype, np.number)

tests/load/test_load_satellite.py

@@ -0,0 +1,17 @@
+from ocf_data_sampler.load.satellite import open_sat_data
+import xarray as xr
+import numpy as np
+
+
+def test_open_satellite(sat_zarr_path):
+    da = open_sat_data(zarr_path=sat_zarr_path)
+
+    assert isinstance(da, xr.DataArray)
+    assert da.dims == ("time_utc", "channel", "x_geostationary", "y_geostationary")
+    # 576 is 2 days of data at 5 minutes intervals, 12 * 24 * 2
+    # There are 11 channels
+    # There are 49 x 20 pixels
+    assert da.shape == (576, 11, 49, 20)
+    assert np.issubdtype(da.dtype, np.number)
+
+