CMIP7-data-request-api 1.1.2__py3-none-any.whl

CMIP7_data_request_api-1.1.2.dist-info/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024-2025 CMIP IPO
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

CMIP7_data_request_api-1.1.2.dist-info/METADATA

@@ -0,0 +1,210 @@
+Metadata-Version: 2.2
+Name: CMIP7_data_request_api
+Version: 1.1.2
+Summary: A programatic interface to the CMIP7 Data Request
+Author: James Anstey, Gaëlle Rigoudy, Martin Schupfner, Matthew Mizielinski, Daniel Ellis, Elisabeth Dingley, Marie-Pierre Moine
+Project-URL: Homepage, https://github.com/CMIP-Data-Request/CMIP7_DReq_Software
+Project-URL: Issues, https://github.com/CMIP-Data-Request/CMIP7_DReq_Software/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pooch
+Requires-Dist: openpyxl
+Requires-Dist: requests
+Requires-Dist: beautifulsoup4
+
+
+# CMIP7 Data Request Software
+
+This repository contains python code to interact with the [CMIP7 data request](https://wcrp-cmip.org/cmip7/cmip7-data-request/). 
+Its aim is to provide an API and scripts that can produce lists of the variables requested for each CMIP7 experiment, information about the requested variables, and in general allow users to query and utilize the information in the data request.
+
+
+## v1.1 release
+
+The latest **official release** of the Data Request (30 Jan 2025) is tagged as `v1.1`. 
+Access all information about the v1.1 release [on the CMIP website](https://wcrp-cmip.org/cmip7-data-request-v1-1/).
+Those trying out the Software should use:
+- the `v1.1` tag, or
+- the latest stable version, which will be the most recent commmit on the `main` branch.
+
+For the **Quick Start** guide, please see below.
+The **Overview** section explains more about what the Software is intended to do and how this relates to the Content of the data request.
+
+**This Software is under active development** and will continue to evolve following the `v1.1` release. 
+Accordingly we encourage users to try the latest stable version in order to access the latest features.
+
+The next sections provide a brief overview of the Software and explain how to get started.
+While the Software is a work in progress, **the Data Request Task Team encourages user feedback to help us improve upcoming versions.**
+Here are some ways to provide feedback:
+- For specific questions or issues (such as bugs) please [open a github issue](https://github.com/CMIP-Data-Request/CMIP7_DReq_Software/issues).
+- For more general questions or concerns, such as suggestions for new features, contribute to the Software's [github discussion forum](https://github.com/CMIP-Data-Request/CMIP7_DReq_Software/discussions).
+
+
+## Overview
+
+The CMIP7 data request **Software** and **Content** are version controlled in separate github repositories.
+Official releases of the data request correspond to a tag in each of these repositories (e.g., `v1.1`). 
+However the Software can interact with different versions of the Content - for example, to examine changes that have occurred when a new version of the data request is issued.
+
+The data request **Content**, which is version controlled [here](https://github.com/CMIP-Data-Request/CMIP7_DReq_Content), refers to all of the information comprising the data request. 
+This includes descriptions of Opportunities and their lists of requested variables, definitions of the variables, etc.
+The Content is stored as a large `json` file, which is read by the data request Software. 
+However users should not interact with this `json` file directly and its structure is not designed for readability.
+Users do not need to manually download the Content as this is done automatically by the Software (see "Getting Started", below, for further details).
+
+The data request Content is an automatic export from **Airtable**, which a cloud platform used by the Data Request Task Team and CMIP IPO to facilitate ongoing community engagement in developing the data request.
+Airtable provides users with a browseable web interface to explore data request information contained in relational databases that are referred to as "bases".
+These Airtable bases contain interlinked tables that constitute the primary source of data request information.
+
+The Content of each official release of the data request can be explored online using the [Airtable interface](https://bit.ly/CMIP7-DReq-v1_0).
+This provides a browseable web view of the Content, allowing users to follow links between different elements of the data request - for example, to view the variables requested by a given Opportunity, or to view the Opportunities that request a given variable.
+This view is complementary to the access to the Content that is provided via the Software, and both access methods (Airtable and Software) are based on the same underlying information.
+
+
+Using the data request **Software** provides a way to interact programmatically with the data request Content, such as to:
+
+- Given a list of supported Opportunities and their priorities, produce lists of variables to output for each experiment (see Getting Started section to test this functionality),
+- Output the CF-compliant metadata characterizing each variable - an example file with some of the metadata for each requested variable is [available since v1.0](https://github.com/CMIP-Data-Request/CMIP7_DReq_Software/tree/main/scripts/variable_info/all_var_info.json),
+- Compare the requested output of CMIP7 experiments to a given model's published CMIP6 output.
+
+An aim of the Software is to facilitate integration of the data request into modelling workflows.
+Suggestions for functionality are welcome in the [github discussion forum](https://github.com/CMIP-Data-Request/CMIP7_DReq_Software/discussions).
+
+
+During development, the Software and Content repositories reside in the github organisation https://github.com/CMIP-Data-Request.
+Stable releases will eventually be migrated into the https://github.com/WCRP-CMIP organisation.
+
+
+## Quick Start
+
+To get started by cloning from github, in a shell session clone the Software and navigate to the `scripts/` directory:
+```bash
+git clone git@github.com:CMIP-Data-Request/CMIP7_DReq_Software.git
+cd CMIP7_DReq_Software
+```
+Alternately, see below for instructions on installing using `pip`, but note the following section's guidance on setting up an environment with the required dependencies.
+
+### Environment setup
+
+The `env.yml` file can be used to create a conda environment in which to run the Software:
+```bash
+conda env create -n my_dreq_env --file env.yml
+```
+where `my_dreq_env` should be replaced with your preferred environment name. 
+Activate this environment:
+```bash
+conda activate my_dreq_env
+```
+(On some systems it may be `source activate my_dreq_env`.)
+Note that `env.yml` explicitly avoids using the conda `defaults` channel.
+
+Alternately, the `requirements.txt` file  can be used to create a python virtual environment:
+```bash
+python -m venv my_dreq_env_dir
+source my_dreq_env_dir/bin/activate
+pip install --upgrade pip
+pip install -r requirements.txt
+```
+replacing `my_dreq_env_dir` with the path where you wish to store your environment.
+This approach avoids using `conda`.
+
+### Running the software
+
+There are two example workflow scripts in the `scripts/` directory. 
+In a shell session where the environment (conda or venv/virtualenv, as explained above) has been activated, running:
+```bash
+python workflow_example.py
+```
+will produce a `json` file listing requested variables for each CMIP7 experiment.
+The same functionality is available from a **command-line interface**, which can be accessed by installing the python package using pip (see below) and then using the `export_dreq_lists_json` command.
+Running:
+```bash
+python workflow_example_2.py
+```
+will produce a set of `csv` files (i.e., spreadsheets) summarizing different aspects of the data request content.
+
+
+## Pip installation
+
+If you have a conda or virtual (venv, virtualenv) environment which already has the dependencies of
+this package you can install the code using
+```bash
+python -m pip install git+https://github.com/CMIP-Data-Request/CMIP7_DReq_Software.git@<tag>
+```
+where `<tag>` needs to be replaced with the version you wish to
+install.
+
+If installation is successful you should be able to run the command 
+```bash
+export_dreq_lists_json --all_opportunities v1.1 amip.json --experiments amip
+```
+to confirm that the variable list for the amip experiment can be produced.
+Multiple experiments can be specified as a space-delimited list of case-sensitive experiment names (e.g., `--experiments amip historical piControl`).
+Omitting the `--experiments` argument is equivalent to including all experiments in the data request.
+To produce variable lists that support only a subset of Opportunities, use the `--opportunities_file` argument by running:
+```bash
+export_dreq_lists_json --opportunities_file opportunities.json v1.1 output.json
+```
+This will generate the `opportunities.json` template file in which Opportunity support can be specified (note that the list of available Opportunities may be updated in newer data request versions).
+Edit this file to by setting unsupported Opportunities to `false` and then re-run the exact same command as above to generate the output file with lists of requested variables.
+
+The package can be uninstalled using
+```bash
+python -m pip uninstall CMIP7_data_request_api
+```
+
+
+
+### Development
+
+To install from a local copy for development purposes cd to the root
+of the repository and run
+```bash
+python -m pip install -e .
+```
+
+Command line utilities should be hosted under the
+`data_request_api.command_line` package and pointed at by adding
+references to the appropriate `main()` routine into the 
+`[project:scripts]` section of the pyproject.toml file.
+
+## Further details
+
+Th example script and the command-line tool contain a workflow to access the data request Content, specify a list of Opportunities and priority levels of variables, and output the lists of variables requested from each experiment in the specified Opportunities.
+An example of the json file produced by running this script, which contains the names of output variables requested for each experiment, is available in `scripts/examples/`.
+The example output file assumes that all data request Opportunities are supported at all priority levels, but this choice can be modified by the user.
+
+Each listed variable in the output file is currently identified by a unique "compound name" using CMIP6-era table names and short variable names (`Amon.tas`, `Omon.tos`, etc).
+Variable names may change in upcoming releases, but in any case a mapping to CMIP6-era variable names will be retained in the data request so as to allow comparison with CMIP6 output (for those variables that were defined in CMIP6).
+
+To access the data request Content, the example script first needs to identify the version of the data request Content that is being used. 
+This is done in the example script by specifying a tag in the Content repo and calling the retrieval function.
+For example:
+```python
+dc.retrieve('v1.1')
+```
+downloads `v1.1` of the Content into local cache, if it is not already there.
+The script can then access it by loading it into a python dict variable:
+```python
+content = dc.load('v1.1')
+```
+Currently a single version of the Content `json` file for a versioned release is roughly 20 MB in size.
+The size of local cache can be managed by deleting unused versions.
+For example, to remove a specific version:
+```python
+dc.delete('v1.1')
+```
+Or to remove all locally cached versions:
+```python
+dc.delete()
+```
+
+## Contributors
+
+[![Contributors](https://contrib.rocks/image?repo=CMIP-Data-Request/CMIP7_DReq_Software)](https://github.com/CMIP-Data-Request/CMIP7_DReq_Software/graphs/contributors/)
+
+Thanks to our contributors!

CMIP7_data_request_api-1.1.2.dist-info/RECORD

@@ -0,0 +1,36 @@
+data_request_api/__init__.py,sha256=4-ghPak2S6zfMqmnlxW2GCgPb5s79znpKa2hGEGXcE4,24
+data_request_api/version.py,sha256=nmVMEP2kpLDIzjgS-U1taCdJbVrmv_k8ao6RojGcvRg,411
+data_request_api/command_line/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+data_request_api/command_line/export_dreq_lists_json.py,sha256=NfAMd86Zm_MwSPpODhLEloTx-ZXOKyW7NrkcR3SQue8,6230
+data_request_api/dev/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+data_request_api/dev/JA/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+data_request_api/dev/JA/check_plev_requests.py,sha256=1DSOEM4ZqREVgWwjS3vnBc_cRNVDMGaHD8NTJfrMTwI,16978
+data_request_api/dev/JA/read_feedback_spreadsheet.py,sha256=424hz7ti-YUcvZzOxvnGL-nqTY-dVvkIrEPszR3rFfE,5334
+data_request_api/dev/JA/workflow_example_GRtest.py,sha256=bywNLFXCEMv5ALoU15UfUFWtNpBZlQ-tieD34nIA0Co,9536
+data_request_api/dev/JA/workflow_example_test.py,sha256=fMcAsgpgFRhaWZrmUjHxQHBk4QHp_h_EZPEdYdEvxB8,6616
+data_request_api/dev/MM/checksum.py,sha256=_ZICl46LerjVSSAspS6mrOWOXA6liyDoDeBur5pYMsE,1884
+data_request_api/dev/MM/walking_data_request.ipynb,sha256=urrmicRTPrkQhyy8Ux4w0l-2Nx7Dlg35jWi-RCwaf-U,13457
+data_request_api/dev/MS/dreq_content_and_walking_data_request.ipynb,sha256=51Ry2vy7dbK6lLFsO4fjldXdR3TMP3zTzdOpA9lQGNI,142409
+data_request_api/stable/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+data_request_api/stable/content/README.MD,sha256=Fl7NpNlxEAP4xXTENPXLyPavaRfHsz04akGgW87aQ8I,3884
+data_request_api/stable/content/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+data_request_api/stable/content/dump_transformation.py,sha256=yWkQN4nvICrTcm08ISsR5DTKssl6EWd-Kpv1wKruJ6M,32211
+data_request_api/stable/content/dreq_api/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+data_request_api/stable/content/dreq_api/consolidate_export.py,sha256=ZGaYqKPibA8XkrItIgZpbbcZJ8a08RO2kLqbsNeo59g,21382
+data_request_api/stable/content/dreq_api/dreq_content.py,sha256=LWawxzlxr6_1Fe4IvNyUayu1pea2s_3hy4RIBMiAMcs,20459
+data_request_api/stable/content/dreq_api/mapping_table.py,sha256=Q5vfRKCgQCMGll1hCYFg7qaFBs-I3fI9toEMSF5znDk,12241
+data_request_api/stable/content/dreq_api/test_dreq_content.py,sha256=JqktWhOAoCXTj3_ASQy3n64pZ9idv_QdCdU9lWuELlA,6488
+data_request_api/stable/query/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+data_request_api/stable/query/data_request.py,sha256=DazgdQ815Agwv_Cz2ypzBgLIHZtN7n2Yhi4rQleU3jw,48219
+data_request_api/stable/query/dreq_classes.py,sha256=ineO9CnHU0QsTys8_DNuPfQ_wkHeUIUxbK7kWsp1KBY,16280
+data_request_api/stable/query/dreq_query.py,sha256=qrd3F9b4rkYCNEA89BU-iqBwztohSO8TdIypPyH7w0A,40269
+data_request_api/stable/query/vocabulary_server.py,sha256=uB27mnG9yy1zRykDt1dwymXStxhcHLW29uk3Imbddzo,7193
+data_request_api/stable/utilities/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+data_request_api/stable/utilities/logger.py,sha256=85qGdfVa5hsssh-rseiQP1c_ABvQ22b81njvjFRLYUY,1772
+data_request_api/stable/utilities/tools.py,sha256=6OLuZoKeQn_LTb5Qx6548jh0JvDXCLk7IbE9gw_zXDc,1408
+CMIP7_data_request_api-1.1.2.dist-info/LICENSE,sha256=eQ82CZRtA1kendI-gjZRrxjYdX8WqS8m2OsHhz2ayqw,1070
+CMIP7_data_request_api-1.1.2.dist-info/METADATA,sha256=QvqLDsZYLP9mJFD52goC9raKiAmXtoHuoSC-s7MvSHs,11691
+CMIP7_data_request_api-1.1.2.dist-info/WHEEL,sha256=In9FTNxeP60KnTkGw7wk6mJPYd_dQSjEZmXdBdMCI-8,91
+CMIP7_data_request_api-1.1.2.dist-info/entry_points.txt,sha256=L4ZgviM_N485tZsGXAg8HJKCedfN9QVTckgf_wARuk4,101
+CMIP7_data_request_api-1.1.2.dist-info/top_level.txt,sha256=9pmO7hEsU_5ytqR82jRH01BvuSlLC2_dRNG6izwsOWE,17
+CMIP7_data_request_api-1.1.2.dist-info/RECORD,,

CMIP7_data_request_api-1.1.2.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (75.8.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

CMIP7_data_request_api-1.1.2.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+export_dreq_lists_json = data_request_api.command_line.export_dreq_lists_json:main

CMIP7_data_request_api-1.1.2.dist-info/top_level.txt

@@ -0,0 +1 @@
+data_request_api

data_request_api/__init__.py

@@ -0,0 +1 @@
+from .version import *

data_request_api/command_line/export_dreq_lists_json.py

@@ -0,0 +1,136 @@
+#!/usr/bin/env python
+"""
+Command line interface for retrieving simple variable lists from the data request.
+"""
+
+import sys
+import json
+import os
+import argparse
+from collections import OrderedDict
+
+# import data_request_api.stable.content.dreq_api as dreq_api
+import data_request_api
+import data_request_api.stable.content.dreq_api.dreq_content as dc
+import data_request_api.stable.query.dreq_query as dq
+
+
+def parse_args():
+    """
+    Parse command line arguments
+    """
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument('dreq_version', choices=dc.get_versions(), help="data request version")
+    parser.add_argument('--opportunities_file', type=str, help="path to JSON file listing opportunities to respond to. If it doesn't exist a template will be created")
+    parser.add_argument('--all_opportunities', action='store_true', help="respond to all opportunities")
+    parser.add_argument('--experiments', nargs='+', type=str, help='limit output to the specified experiments (space-delimited list, case sensitive)')
+    parser.add_argument('--priority_cutoff', default='low', choices=dq.PRIORITY_LEVELS, help="discard variables that are requested at lower priority than this cutoff priority")
+    parser.add_argument('output_file', help='file to write JSON output to')
+    parser.add_argument('--version', action='store_true', help='Return version information and exit')
+    return parser.parse_args()
+
+
+def main():
+    """
+    main routine
+    """
+    args = parse_args()
+
+    if args.version:
+        print("CMIP7 data request api version {}".format(data_request_api.version))
+        sys.exit(0)
+    use_dreq_version = args.dreq_version
+
+    # Download specified version of data request content (if not locally cached)
+    dc.retrieve(use_dreq_version)
+    # Load content into python dict
+    content = dc.load(use_dreq_version)
+    # Render data request tables as dreq_table objects
+    base = dq.create_dreq_tables_for_request(content)
+
+    # Deal with opportunities
+    if args.opportunities_file:
+        opportunities_file = args.opportunities_file
+        Opps = base['Opportunity']
+        if not os.path.exists(opportunities_file):
+            # create opportunities file template
+            use_opps = sorted([opp.title for opp in Opps.records.values()], key=str.lower)
+            default_opportunity_dict = OrderedDict({
+                'Header' : OrderedDict({
+                    'Description' : 'Opportunities template file for use with export_dreq_lists_json. Set supported/unsupported Opportunities to true/false.',
+                    'dreq content version': use_dreq_version,
+                    'dreq api version' : data_request_api.version,
+                }),
+                'Opportunity' : OrderedDict({title : True for title in use_opps})
+            })
+            with open(opportunities_file, 'w') as fh:
+                json.dump(default_opportunity_dict, fh, indent=4)
+                print("written opportunities dict to {}. Please edit and re-run".format(opportunities_file))
+                sys.exit(0)
+        else:
+            # load existing opportunities file
+            with open(opportunities_file, 'r') as fh:
+                opportunity_dict = json.load(fh)
+            
+            dreq_version = opportunity_dict['Header']['dreq content version']
+            if dreq_version != use_dreq_version:
+                raise ValueError('Data request version mismatch!' + \
+                                 f'\nOpportunities file was generated for data request version {dreq_version}' + \
+                                 f'\nPlease regenerate the file using version {use_dreq_version}')
+
+            opportunity_dict = opportunity_dict['Opportunity']
+
+            # validate opportunities
+            # (mismatches can occur if an opportunities file created with an earlier data request version is loaded)
+            valid_opps = [opp.title for opp in Opps.records.values()]
+            invalid_opps = [title for title in opportunity_dict if title not in valid_opps]
+            if invalid_opps:
+                raise ValueError(f'\nInvalid opportunities were found in {opportunities_file}:\n' + '\n'.join(sorted(invalid_opps, key=str.lower)))
+
+            # filter opportunities
+            use_opps = [title for title in opportunity_dict if opportunity_dict[title]]
+
+    elif args.all_opportunities:
+        use_opps = 'all'
+    else:
+        print("Please use one of the opportunities arguments")
+        sys.exit(1)
+
+    # Get consolidated list of requested variables that supports these opportunities
+    dq.DREQ_VERSION = use_dreq_version
+    expt_vars = dq.get_requested_variables(base, use_opps, priority_cutoff=args.priority_cutoff, verbose=False)
+
+    # filter output by requested experiments
+    if args.experiments:
+        experiments = list(expt_vars['experiment'].keys()) # names of experiments requested by opportunities in use_opps
+
+        # validate the requested experiment names
+        Expts = base['Experiments']
+        valid_experiments = [expt.experiment for expt in Expts.records.values()] # all valid experiment names in data request
+        invalid_experiments = [entry for entry in args.experiments if entry not in valid_experiments]
+        if invalid_experiments:
+            raise ValueError('\nInvalid experiments: ' + ', '.join(sorted(invalid_experiments, key=str.lower)) + \
+                '\nValid experiment names: ' + ', '.join(sorted(valid_experiments, key=str.lower)))
+
+        # discard experiments that aren't requested
+        for entry in experiments:
+            if entry not in args.experiments:
+                del expt_vars['experiment'][entry]
+
+    # Construct output
+    if len(expt_vars['experiment']) > 0:
+
+        # Show user what was found
+        dq.show_requested_vars_summary(expt_vars, use_dreq_version)
+
+        # Write json file with the variable lists
+        content_path = dc._dreq_content_loaded['json_path']
+        outfile = args.output_file
+        dq.write_requested_vars_json(outfile, expt_vars, use_dreq_version, args.priority_cutoff, content_path)
+
+    else:
+        print(f'\nFor data request version {use_dreq_version}, no requested variables were found')
+
+if __name__ == '__main__':
+    main()