CMIP7-data-request-api 1.2b2__tar.gz → 1.2.1__tar.gz

{cmip7_data_request_api-1.2b2 → cmip7_data_request_api-1.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: CMIP7_data_request_api
-Version: 1.2b2
+Version: 1.2.1
 Summary: A programmatic interface to the CMIP7 Data Request
 Author: James Anstey, Gaëlle Rigoudy, Martin Schupfner, Matthew Mizielinski, Daniel Ellis, Elisabeth Dingley, Marie-Pierre Moine
 License-Expression: MIT
@@ -20,6 +20,7 @@ Dynamic: license-file
 
 [![pypi](https://img.shields.io/pypi/v/CMIP7-data-request-api.svg)](https://pypi.python.org/pypi/CMIP7-data-request-api)
 [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/CMIP-Data-Request/CMIP7_DReq_Software/main?filepath=notebooks)
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/CMIP-Data-Request/CMIP7_DReq_Software/)
 [![NBviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.jupyter.org/github/CMIP-Data-Request/CMIP7_DReq_Software/tree/main/notebooks/)
 [![license](https://img.shields.io/github/license/CMIP-Data-Request/CMIP7_DReq_Software.svg)](https://github.com/CMIP-Data-Request/CMIP7_DReq_Software/blob/main/LICENSE)
 [![status](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)
@@ -29,7 +30,7 @@ Dynamic: license-file
 Python software to interact with the [CMIP7 data request](https://wcrp-cmip.org/cmip7/cmip7-data-request/).
 It provides an API to query and utilize the information in the data request, including example scripts and notebooks showing how to use the API.
 
-For the **Quick Start** guide, please see below.
+Please see below for the [installation guide](#installation) or how to [try it without installation](#try-it-without-installation).
 
 The Data Request Task Team encourages user feedback to help us improve the software.
 Here are some ways to provide feedback:
@@ -37,22 +38,25 @@ Here are some ways to provide feedback:
 - For more general questions or concerns, such as suggestions for new features, contribute to the Software's [github discussion forum](https://github.com/CMIP-Data-Request/CMIP7_DReq_Software/discussions).
 
 
-## v1.2 Data Request release
+## v1.2.1 Data Request release
 
-The latest **official release** of the CMIP7 Data Request is `v1.2` (31 March 2025)
-Access all information about the v1.2 release [on the CMIP website](https://wcrp-cmip.org/cmip7-data-request-v1-2/).
+The latest **official release** of the CMIP7 Data Request is `v1.2.1` (26 April 2025).
+[Learn more about this release on the CMIP website](https://wcrp-cmip.org/cmip7-data-request-v1-2-1/).
 
-Please note that a technical update to the Data Request, `v1.2.1`, is planned for late April 2025.
-This update will affect some details such as the names of CMOR variables, without substantially changing the scientific content of the request.
-For further information please see the [v1.2 release page](https://wcrp-cmip.org/cmip7-data-request-v1-2/).
-An update of this software will follow the release of `v1.2.1`.
+
+## Try It Without Installation
+
+You can launch and interact with this repository in a live environment via [Binder](https://mybinder.org/) or [Google Colab](https://colab.research.google.com/) — no installation needed, just click on one of the badges to run it in your browser:
+
+[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/CMIP-Data-Request/CMIP7_DReq_Software/main?filepath=notebooks)
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/CMIP-Data-Request/CMIP7_DReq_Software/)
 
 
 ## Installation
 
 ### Quick Start
 
-In a python virtual environment that already has the dependencies installed, do:
+In a python virtual environment or conda environment in which you want to install the package, do:
 
 ```bash
 pip install CMIP7-data-request-api
@@ -64,9 +68,14 @@ If an environment first needs to be created, you can do:
 python -m venv my_dreq_env
 source my_dreq_env/bin/activate
 pip install --upgrade pip
+pip install CMIP7-data-request-api
+```
+
+This will automatically install the dependencies, but if necessary they can be manually installed by doing:
+
+```bash
 wget https://raw.githubusercontent.com/CMIP-Data-Request/CMIP7_DReq_Software/refs/heads/main/requirements.txt
 pip install -r requirements.txt 
-pip install CMIP7-data-request-api
 ```
 
 using the `requirements.txt` file from the top-level directory of this repository, which lists the package dependencies, and `my_dreq_env` can be changed to whatever environment name is preferred.

{cmip7_data_request_api-1.2b2 → cmip7_data_request_api-1.2.1}/README.md

@@ -1,5 +1,6 @@
 [![pypi](https://img.shields.io/pypi/v/CMIP7-data-request-api.svg)](https://pypi.python.org/pypi/CMIP7-data-request-api)
 [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/CMIP-Data-Request/CMIP7_DReq_Software/main?filepath=notebooks)
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/CMIP-Data-Request/CMIP7_DReq_Software/)
 [![NBviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.jupyter.org/github/CMIP-Data-Request/CMIP7_DReq_Software/tree/main/notebooks/)
 [![license](https://img.shields.io/github/license/CMIP-Data-Request/CMIP7_DReq_Software.svg)](https://github.com/CMIP-Data-Request/CMIP7_DReq_Software/blob/main/LICENSE)
 [![status](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)
@@ -9,7 +10,7 @@
 Python software to interact with the [CMIP7 data request](https://wcrp-cmip.org/cmip7/cmip7-data-request/).
 It provides an API to query and utilize the information in the data request, including example scripts and notebooks showing how to use the API.
 
-For the **Quick Start** guide, please see below.
+Please see below for the [installation guide](#installation) or how to [try it without installation](#try-it-without-installation).
 
 The Data Request Task Team encourages user feedback to help us improve the software.
 Here are some ways to provide feedback:
@@ -17,22 +18,25 @@ Here are some ways to provide feedback:
 - For more general questions or concerns, such as suggestions for new features, contribute to the Software's [github discussion forum](https://github.com/CMIP-Data-Request/CMIP7_DReq_Software/discussions).
 
 
-## v1.2 Data Request release
+## v1.2.1 Data Request release
 
-The latest **official release** of the CMIP7 Data Request is `v1.2` (31 March 2025)
-Access all information about the v1.2 release [on the CMIP website](https://wcrp-cmip.org/cmip7-data-request-v1-2/).
+The latest **official release** of the CMIP7 Data Request is `v1.2.1` (26 April 2025).
+[Learn more about this release on the CMIP website](https://wcrp-cmip.org/cmip7-data-request-v1-2-1/).
 
-Please note that a technical update to the Data Request, `v1.2.1`, is planned for late April 2025.
-This update will affect some details such as the names of CMOR variables, without substantially changing the scientific content of the request.
-For further information please see the [v1.2 release page](https://wcrp-cmip.org/cmip7-data-request-v1-2/).
-An update of this software will follow the release of `v1.2.1`.
+
+## Try It Without Installation
+
+You can launch and interact with this repository in a live environment via [Binder](https://mybinder.org/) or [Google Colab](https://colab.research.google.com/) — no installation needed, just click on one of the badges to run it in your browser:
+
+[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/CMIP-Data-Request/CMIP7_DReq_Software/main?filepath=notebooks)
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/CMIP-Data-Request/CMIP7_DReq_Software/)
 
 
 ## Installation
 
 ### Quick Start
 
-In a python virtual environment that already has the dependencies installed, do:
+In a python virtual environment or conda environment in which you want to install the package, do:
 
 ```bash
 pip install CMIP7-data-request-api
@@ -44,9 +48,14 @@ If an environment first needs to be created, you can do:
 python -m venv my_dreq_env
 source my_dreq_env/bin/activate
 pip install --upgrade pip
+pip install CMIP7-data-request-api
+```
+
+This will automatically install the dependencies, but if necessary they can be manually installed by doing:
+
+```bash
 wget https://raw.githubusercontent.com/CMIP-Data-Request/CMIP7_DReq_Software/refs/heads/main/requirements.txt
 pip install -r requirements.txt 
-pip install CMIP7-data-request-api
 ```
 
 using the `requirements.txt` file from the top-level directory of this repository, which lists the package dependencies, and `my_dreq_env` can be changed to whatever environment name is preferred.

cmip7_data_request_api-1.2.1/binder/environment.yml

@@ -0,0 +1,13 @@
+channels:
+  - conda-forge
+  - nodefaults
+dependencies:
+  - bs4
+  - coverage
+  - openpyxl
+  - pooch
+  - pytest
+  - pyyaml
+  - requests
+  - pip:
+    - CMIP7-data-request-api

{cmip7_data_request_api-1.2b2 → cmip7_data_request_api-1.2.1}/data_request_api/CMIP7_data_request_api.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: CMIP7_data_request_api
-Version: 1.2b2
+Version: 1.2.1
 Summary: A programmatic interface to the CMIP7 Data Request
 Author: James Anstey, Gaëlle Rigoudy, Martin Schupfner, Matthew Mizielinski, Daniel Ellis, Elisabeth Dingley, Marie-Pierre Moine
 License-Expression: MIT
@@ -20,6 +20,7 @@ Dynamic: license-file
 
 [![pypi](https://img.shields.io/pypi/v/CMIP7-data-request-api.svg)](https://pypi.python.org/pypi/CMIP7-data-request-api)
 [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/CMIP-Data-Request/CMIP7_DReq_Software/main?filepath=notebooks)
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/CMIP-Data-Request/CMIP7_DReq_Software/)
 [![NBviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.jupyter.org/github/CMIP-Data-Request/CMIP7_DReq_Software/tree/main/notebooks/)
 [![license](https://img.shields.io/github/license/CMIP-Data-Request/CMIP7_DReq_Software.svg)](https://github.com/CMIP-Data-Request/CMIP7_DReq_Software/blob/main/LICENSE)
 [![status](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)
@@ -29,7 +30,7 @@ Dynamic: license-file
 Python software to interact with the [CMIP7 data request](https://wcrp-cmip.org/cmip7/cmip7-data-request/).
 It provides an API to query and utilize the information in the data request, including example scripts and notebooks showing how to use the API.
 
-For the **Quick Start** guide, please see below.
+Please see below for the [installation guide](#installation) or how to [try it without installation](#try-it-without-installation).
 
 The Data Request Task Team encourages user feedback to help us improve the software.
 Here are some ways to provide feedback:
@@ -37,22 +38,25 @@ Here are some ways to provide feedback:
 - For more general questions or concerns, such as suggestions for new features, contribute to the Software's [github discussion forum](https://github.com/CMIP-Data-Request/CMIP7_DReq_Software/discussions).
 
 
-## v1.2 Data Request release
+## v1.2.1 Data Request release
 
-The latest **official release** of the CMIP7 Data Request is `v1.2` (31 March 2025)
-Access all information about the v1.2 release [on the CMIP website](https://wcrp-cmip.org/cmip7-data-request-v1-2/).
+The latest **official release** of the CMIP7 Data Request is `v1.2.1` (26 April 2025).
+[Learn more about this release on the CMIP website](https://wcrp-cmip.org/cmip7-data-request-v1-2-1/).
 
-Please note that a technical update to the Data Request, `v1.2.1`, is planned for late April 2025.
-This update will affect some details such as the names of CMOR variables, without substantially changing the scientific content of the request.
-For further information please see the [v1.2 release page](https://wcrp-cmip.org/cmip7-data-request-v1-2/).
-An update of this software will follow the release of `v1.2.1`.
+
+## Try It Without Installation
+
+You can launch and interact with this repository in a live environment via [Binder](https://mybinder.org/) or [Google Colab](https://colab.research.google.com/) — no installation needed, just click on one of the badges to run it in your browser:
+
+[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/CMIP-Data-Request/CMIP7_DReq_Software/main?filepath=notebooks)
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/CMIP-Data-Request/CMIP7_DReq_Software/)
 
 
 ## Installation
 
 ### Quick Start
 
-In a python virtual environment that already has the dependencies installed, do:
+In a python virtual environment or conda environment in which you want to install the package, do:
 
 ```bash
 pip install CMIP7-data-request-api
@@ -64,9 +68,14 @@ If an environment first needs to be created, you can do:
 python -m venv my_dreq_env
 source my_dreq_env/bin/activate
 pip install --upgrade pip
+pip install CMIP7-data-request-api
+```
+
+This will automatically install the dependencies, but if necessary they can be manually installed by doing:
+
+```bash
 wget https://raw.githubusercontent.com/CMIP-Data-Request/CMIP7_DReq_Software/refs/heads/main/requirements.txt
 pip install -r requirements.txt 
-pip install CMIP7-data-request-api
 ```
 
 using the `requirements.txt` file from the top-level directory of this repository, which lists the package dependencies, and `my_dreq_env` can be changed to whatever environment name is preferred.

{cmip7_data_request_api-1.2b2 → cmip7_data_request_api-1.2.1}/data_request_api/CMIP7_data_request_api.egg-info/SOURCES.txt

@@ -10,6 +10,7 @@ requirements.txt
 .github/workflows/pip_install.yml
 .github/workflows/publish2pypi.yml
 .github/workflows/updates.yml
+binder/environment.yml
 data_request_api/.DS_Store
 data_request_api/CMIP7_data_request_api.egg-info/PKG-INFO
 data_request_api/CMIP7_data_request_api.egg-info/SOURCES.txt
@@ -131,9 +132,11 @@ notebooks/workflow_example.ipynb
 notebooks/outdir/.gitkeep
 scripts/check_consolidation.py
 scripts/check_consolidation2.py
+scripts/check_consolidation_and_transformation.py
 scripts/check_variables_attributes.py
 scripts/database_transformation.py
 scripts/workflow_example.py
 scripts/workflow_example_2.py
-scripts/examples/requested_v1.2.json
-scripts/examples/variables_v1.2.json
+scripts/examples/requested_v1.2.1.json
+scripts/examples/variables_v1.2.1.json
+scripts/sandbox/create_brand_name.py

{cmip7_data_request_api-1.2b2 → cmip7_data_request_api-1.2.1}/data_request_api/data_request_api/command_line/config.py

@@ -8,6 +8,7 @@ from data_request_api.utilities import config as dreqcfg
 def main():
     parser = argparse.ArgumentParser(
         description="Config CLI",
+        formatter_class=argparse.RawTextHelpFormatter,
         epilog="""Arguments:
   init (or no arguments): Initialize the config file,
       i.e. create a config file with default values if it does not exist.
@@ -17,8 +18,8 @@ def main():
   help: print this help message.
 
 Examples:
-  python -m utilities.config offline true
-  python -m utilities.config reset""",
+  CMIP7_data_request_api_config offline true
+  CMIP7_data_request_api_config reset""",
     )
 
     if parser.prog.startswith("config"):
@@ -38,7 +39,7 @@ Examples:
         dreqcfg.CONFIG_FILE = args.cfgfile
 
     if len(args.command) == 0 or args.command[0] == "init":
-        print("Initializing config file:")
+        print(f"Initializing config file: '{dreqcfg.CONFIG_FILE}'")
         dreqcfg.load_config()
     elif args.command[0] == "reset":
         print("Resetting config with defaults:")

{cmip7_data_request_api-1.2b2 → cmip7_data_request_api-1.2.1}/data_request_api/data_request_api/command_line/estimate_dreq_volume.py

@@ -64,7 +64,7 @@ def get_variable_size(var_info, dreq_dim_sizes, time_dims, freq_times_per_year,
                 n = 24 * 12
             else:
                 n = freq_times_per_year[frequency]
-            temporal_shape = dim
+            temporal_shape = time_dims[dim]
         elif dim in config['dimensions']:
             # Use model-specific dimension size
             n = config['dimensions'][dim]
@@ -74,7 +74,7 @@ def get_variable_size(var_info, dreq_dim_sizes, time_dims, freq_times_per_year,
             n = dreq_dim_sizes[dim]
 
         if n is None:
-            raise ValueError(f'No size found for dimension {dim}')
+            raise ValueError(f'No size found for dimension: {dim}')
 
         dim_sizes[dim] = n
 
@@ -101,7 +101,7 @@ def parse_args():
     parser.add_argument('request', type=str,
                         help='json file specifying variables requested by experiment' +
                         ' (output from export_dreq_lists_json, which specifies the data request version)' +
-                        ' OR can be a data request version (e.g. "v1.2") if using ')
+                        ' OR can be a data request version (e.g. "v1.2")')
     # Optional arguments
     parser.add_argument('-v', '--variables', nargs='+', type=str,
                         help='include only the specified variables in the estimate')
@@ -167,12 +167,13 @@ years: 1
         config = yaml.safe_load(f)
         print('Loaded ' + config_file)
 
-    warning_msg = '\n * * * WARNING * * *'
-    warning_msg += '\n These volumes are an initial estimate, to be improved in the next software version.'
-    warning_msg += '\n They should be used with caution and verified against known data volumes.\n'
+    warning_msg = '\n * * * * * * * * * * * * * * * WARNING * * * * * * * * * * * * * * *'
+    warning_msg += '\n These volumes are an initial estimate.'
+    warning_msg += '\n They should be used with caution and verified against known data volumes.'
+    warning_msg += '\n * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *\n'
 
     request_from_input_file = None
-    if os.path.exists(args.request):
+    if os.path.isfile(args.request):
         # Argument is a file that lists requested variables
         filepath = args.request
         with open(filepath, 'r') as f:
@@ -232,8 +233,20 @@ years: 1
     if set(freq_times_per_year.keys()) != set(freqs):
         raise Exception('Times per year must be defined for all available frequencies')
 
-    # Get available time dimensions ('time-intv', 'time-point', etc)
-    time_dims = [rec.name for rec in dreq_tables['temporal shape'].records.values()]
+    # Get mapping from time dimension name to temporal shape name
+    # {'time1': 'time-point', 'time': 'time-intv', ... etc}
+    time_dims = {}
+    for rec in dreq_tables['temporal shape'].records.values():
+        shape_name = rec.name
+        if hasattr(rec, 'dimensions'):
+            assert len(rec.dimensions) == 1
+            link = rec.dimensions[0]
+            dim_rec = dreq_tables['coordinates and dimensions'].get_record(link)
+            dim_name = dim_rec.name
+        else:
+            dim_name = 'None'
+        assert dim_name not in time_dims, 'time dimension names are not unique'
+        time_dims[dim_name] = shape_name
 
     # Get metadata for variables
     variables = dq.get_variables_metadata(
@@ -307,7 +320,11 @@ years: 1
     for expt in expts:
         expt_rec = expt_records[expt]
 
-        num_years = expt_rec.size_years_minimum
+        if hasattr(expt_rec, 'size_years_minimum'):
+            num_years = expt_rec.size_years_minimum
+        else:
+            num_years = 100
+            print(f'Warning: number of years not found for experiment {expt}, assuming size_years_minimum = {num_years}')
         num_ensem = 1
 
         # Loop over priority levels of requeste variables
@@ -339,7 +356,12 @@ years: 1
                     # Assume this is a climatology, so don't multiply by number of years
                     pass
                 else:
-                    assert temporal_shape in ['time-intv', 'time-point', 'monthly-mean-daily-stat'], \
+                    valid_shapes = ['time-intv', 'time-point', 'monthly-mean-daily-stat']
+                    version_tuple = dq.get_dreq_version_tuple(use_dreq_version)
+                    if version_tuple[:2] < (1, 2):
+                        # Prior to v1.2, monthly-mean-daily-stat was called monthly-mean-stat
+                        valid_shapes.append('monthly-mean-stat')
+                    assert temporal_shape in valid_shapes, \
                         'Unknown temporal shape: ' + str(temporal_shape)
                     # Multiply the 1-year size by the minimum number of request years for this experiment
                     size *= num_years

{cmip7_data_request_api-1.2b2 → cmip7_data_request_api-1.2.1}/data_request_api/data_request_api/command_line/export_dreq_lists_json.py

@@ -149,18 +149,19 @@ def main():
 
         # Get metadata for variables
         all_var_info = dq.get_variables_metadata(
-            base,
+            base, use_dreq_version,
             compound_names=all_var_names,
-            use_dreq_version=use_dreq_version  # TO DEPRECATE
+            # use_dreq_version=use_dreq_version  # TO DEPRECATE
         )
 
         # Write output file(s)
         for filepath in args.variables_metadata:
             dq.write_variables_metadata(
                 all_var_info,
+                use_dreq_version,
                 filepath,
                 api_version=data_request_api.version,
-                use_dreq_version=use_dreq_version,
+                # use_dreq_version=use_dreq_version,
                 content_path=dc._dreq_content_loaded['json_path']
             )
 

{cmip7_data_request_api-1.2b2 → cmip7_data_request_api-1.2.1}/data_request_api/data_request_api/content/consolidate_export.py

@@ -122,7 +122,7 @@ def _apply_hard_fixes(data):
     return data
 
 
-def _filter_references(val, key, table, rid):
+def _filter_references(val, key, table, rid, dtype=None):
     """
     Filters lists of or strings with comma-separated references to other records.
     """
@@ -142,8 +142,8 @@ def _filter_references(val, key, table, rid):
                     f"'{table}': Filtered {len(val) - len(filtered)} of {len(val)}"
                     f" references for '{key}' of record '{rid}'."
                 )
-        return filtered
-    elif isinstance(val, str) and "rec" in val:
+        return _fix_dtype(key, filtered, dtype)
+    elif isinstance(val, str) and val.startswith("rec"):
         if "," in val:
             vallist = [v.strip() for v in val.split(",")]
             filtered = [v for v in vallist if v not in filtered_records]
@@ -157,14 +157,14 @@ def _filter_references(val, key, table, rid):
                         f"'{table}': Filtered {len(vallist) - len(filtered)} of"
                         f" {len(vallist)} references for '{key}' of record '{rid}'."
                     )
-            return ",".join(filtered)
+            return _fix_dtype(key, ",".join(filtered), dtype)
         elif val.strip() in filtered_records:
             logger.warning(f"'{table}': Filtered the only reference for '{key}' of record '{rid}'.")
-            return ""
+            return _fix_dtype(key, "", dtype)
         else:
-            return val.strip()
+            return _fix_dtype(key, _fix_str(val), dtype)
     else:
-        return val
+        return _fix_dtype(key, _fix_str(val), dtype)
 
 
 def _gen_rid_uid_map(data):
@@ -194,6 +194,61 @@ def _gen_rid_uid_map(data):
     return rid_uid_map
 
 
+def _fix_str(var):
+    """Adds missing space after commas and strips whitespace from strings."""
+    if isinstance(var, str):
+        return re.sub(r",(?=\S)", ", ", var).strip()
+    else:
+        return var
+
+
+def _fix_str_nested(data):
+    """Adds missing space after commas and strips whitespace from strings in nested dictionary."""
+    sub = re.sub
+    pattern = r",(?=\S)"
+    for table in data.values():
+        for record in table["records"].values():
+            record.update({k: sub(pattern, ", ", v).strip() if isinstance(v, str) else v for k, v in record.items()})
+
+
+def _fix_dtype(fkey, fval, dtype=None):
+    """Fixes data types for record fields."""
+    logger = get_logger()
+
+    if dtype is None:
+        return fval
+    elif dtype == "str":
+        logger.debug(f"Consolidate export: Converting field '{fkey}' ('{fval}') to string.")
+        return str(fval)
+    elif dtype == "int":
+        logger.debug(f"Consolidate export: Converting field '{fkey}' ('{fval}') to int.")
+        return int(fval)
+    elif dtype == "float":
+        logger.debug(f"Consolidate export: Converting field '{fkey}' ('{fval}') to float.")
+        return float(fval)
+    elif dtype == "listofstr":
+        logger.debug(f"Consolidate export: Converting field '{fkey}' to a list of strings.")
+        if isinstance(fval, list):
+            return [str(v) for v in fval]
+        else:
+            return [str(fval)]
+    elif dtype == "listofint":
+        logger.debug(f"Consolidate export: Converting field '{fkey}' to a list of ints.")
+        if isinstance(fval, list):
+            return [int(v) for v in fval]
+        else:
+            return [int(fval)]
+    elif dtype == "listoffloat":
+        logger.debug(f"Consolidate export: Converting field '{fkey}' to a list of floats.")
+        if isinstance(fval, list):
+            return [float(v) for v in fval]
+        else:
+            return [float(fval)]
+    else:
+        logger.warning(f"Consolidate export: Unsupported data type '{dtype}' for field '{fkey}'.")
+        return fval
+
+
 def map_data(data, mapping_table, version, **kwargs):
     """
     Maps the data to the one-base structure using the mapping table.
@@ -310,7 +365,11 @@ def map_data(data, mapping_table, version, **kwargs):
                     "records": {
                         record_id: {
                             mapinfo["internal_consistency"].get(reckey, reckey): _filter_references(
-                                recvalue, reckey, table, record_id
+                                recvalue,
+                                reckey,
+                                table,
+                                record_id,
+                                mapinfo["field_dtypes"].get(mapinfo["internal_consistency"].get(reckey, reckey), None),
                             )
                             for reckey, recvalue in record.items()
                             if reckey not in mapinfo["drop_keys"]
@@ -525,6 +584,9 @@ def map_data(data, mapping_table, version, **kwargs):
         # Consistency fixes
         mapped_data = next(iter(data.values()))
         mapped_data = _apply_consistency_fixes(mapped_data)
+        # String fixes
+        logger.debug("Consolidation: Removing / Adding (un)necessary whitespace to strings.")
+        _fix_str_nested(mapped_data)
         mapped_data["version"] = version
         return {"Data Request": mapped_data}
     else:

{cmip7_data_request_api-1.2b2 → cmip7_data_request_api-1.2.1}/data_request_api/data_request_api/content/dreq_content.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 
-
+import atexit
 import json
 import os
 import re
@@ -65,6 +65,10 @@ except KeyError:
 
 _dreq_content_loaded = {}
 
+# Internal flag used to determine whether a warning on API version can be issued
+# (Purpose is to prevent the warning being issued more than once per session)
+_CHECK_API_VERSION = True
+
 
 def _parse_version(version):
     """Parse a version tag and return a tuple for sorting.
@@ -121,9 +125,6 @@ def get_cached(**kwargs):
                 json_export = _json_raw
             elif kwargs["export"] == "release":
                 json_export = _json_release
-            else:
-                warnings.warn(f"Unknown export type '{kwargs['export']}'. Defaulting to 'release'.")
-                json_export = _json_release
         local_versions = [
             name for name in os.listdir(_dreq_res) if os.path.isfile(os.path.join(_dreq_res, name, json_export))
         ]
@@ -293,6 +294,7 @@ def get_versions(target="tags", **kwargs):
     """
     global versions
     global _versions_retrieved_last
+    global _CHECK_API_VERSION
 
     if target not in ["tags", "branches"]:
         raise ValueError("target must be 'tags' or 'branches'.")
@@ -314,7 +316,14 @@ def get_versions(target="tags", **kwargs):
         if target == "tags" and "dev" not in versions[target]:
             versions[target].append("dev")
 
-    # List tags hosted on GitHub
+    if kwargs['check_api_version'] and not kwargs["offline"]:
+        # Warn user if the API version is not the latest one available on PyPI
+        if _CHECK_API_VERSION:
+            atexit.register(dreqcfg.check_api_version)
+            # Set flag to prevent the same warning being shown more than once
+            _CHECK_API_VERSION = False
+
+    # List tags of dreq versions hosted on GitHub
     return versions[target]
 
 
@@ -393,7 +402,7 @@ def retrieve(version="latest_stable", **kwargs):
     if versions == [None] or not versions:
         raise ValueError(f"Version '{version}' not found.")
     elif version in ["v1.0alpha"] and "export" in kwargs and kwargs["export"] == "raw":
-        warnings.warn(f"For version '{version}' no raw export exists.")
+        warnings.warn(f"For version '{version}' no raw export exists. Defaulting to release export.")
 
     json_paths = dict()
     for version in versions:
@@ -406,8 +415,6 @@ def retrieve(version="latest_stable", **kwargs):
                 json_export = _json_release
             elif kwargs["export"] == "raw":
                 json_export = _json_raw
-            else:
-                warnings.warn(f"Unknown export type '{kwargs['export']}'.")
         elif _version_pattern.match(version):
             json_export = _json_release
         else:
@@ -544,10 +551,6 @@ def delete(version="all", keep_latest=False, **kwargs):
         cached_files = [os.path.join(_dreq_res, v, _json_raw) for v in local_versions]
     elif kwargs["export"] == "release":
         cached_files = [os.path.join(_dreq_res, v, _json_release) for v in local_versions]
-    else:
-        # Since files are to be deleted, not defaulting to "release" but rather
-        #  raising a ValueError
-        raise ValueError(f"Unknown export type '{kwargs['export']}'.")
 
     # Delete files
     for f in cached_files:

{cmip7_data_request_api-1.2b2 → cmip7_data_request_api-1.2.1}/data_request_api/data_request_api/content/dump_transformation.py

@@ -290,11 +290,12 @@ def tidy_content(content, record_to_uid_index):
                     to_remove_entries[subelt][(record_id, uid)] += 1
                 content_string = tmp_content_string
         content[content_subelt] = json.loads(content_string)
-    if "opportunities" in to_remove_entries:
-        to_remove = [elt for (elt, nb) in to_remove_entries["opportunities"].items() if nb == len_list_content]
-        for record_id, _ in to_remove:
-            del record_to_uid_index["opportunities"][record_id]
-        del to_remove_entries["opportunities"]
+    for content_subelt in ["opportunities", "coordinates_and_dimensions"]:
+        if content_subelt in to_remove_entries:
+            to_remove = [elt for (elt, nb) in to_remove_entries[content_subelt].items() if nb == len_list_content]
+            for record_id, _ in to_remove:
+                del record_to_uid_index[content_subelt][record_id]
+            del to_remove_entries[content_subelt]
     for (subelt, to_remove) in to_remove_entries.items():
         to_remove = [elt for (elt, nb) in to_remove.items() if nb == len_list_content]
         for (record_id, uid) in to_remove: