esgf-pyclient 0.3.2__py3-none-any.whl

esgf_pyclient-0.3.2.dist-info/METADATA

@@ -0,0 +1,119 @@
+Metadata-Version: 2.4
+Name: esgf-pyclient
+Version: 0.3.2
+Summary: A library interacting with ESGF services within Python
+Home-page: http://esgf-pyclient.readthedocs.org
+Download-URL: https://github.com/ESGF/esgf-pyclient
+Author: Ag Stephens
+Author-email: Ag.Stephens@stfc.ac.uk
+License: BSD
+Classifier: Development Status :: 4 - Beta
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Topic :: Scientific/Engineering
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Python: >=3.11.0
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: defusedxml
+Requires-Dist: jinja2
+Requires-Dist: requests
+Requires-Dist: requests_cache
+Requires-Dist: webob
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: flake8; extra == "dev"
+Requires-Dist: defusedxml; extra == "dev"
+Requires-Dist: jinja2; extra == "dev"
+Requires-Dist: requests; extra == "dev"
+Requires-Dist: requests_cache; extra == "dev"
+Requires-Dist: webob; extra == "dev"
+Requires-Dist: myproxyclient>=2.1.1; extra == "dev"
+Requires-Dist: nbval; extra == "dev"
+Requires-Dist: sphinx; extra == "dev"
+Requires-Dist: sphinx_mdinclude; extra == "dev"
+Requires-Dist: nbsphinx; extra == "dev"
+Requires-Dist: pandoc; extra == "dev"
+Requires-Dist: nbconvert; extra == "dev"
+Requires-Dist: ipykernel; extra == "dev"
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: download-url
+Dynamic: home-page
+Dynamic: license
+Dynamic: license-file
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+[![Documentation Status](https://img.shields.io/badge/docs-latest-brightgreen.svg)](https://esgf-pyclient.readthedocs.io/en/latest/?badge=latest)
+[![Build Status](https://github.com/ESGF/esgf-pyclient/workflows/build%20%E2%9A%99%EF%B8%8F/badge.svg)](https://github.com/ESGF/esgf-pyclient/actions)
+[![GitHub license](https://img.shields.io/github/license/ESGF/esgf-pyclient.svg)](https://github.com/ESGF/esgf-pyclient/blob/master/LICENSE)
+
+# esgf-pyclient
+
+ESGF PyClient is a Python package designed for interacting with the [Earth System Grid Federation](https://esgf.llnl.gov/) system.
+
+> [!WARNING]
+> This package interacts with the
+> [legacy ESGF Search API](https://esgf.github.io/esg-search/ESGF_Search_RESTful_API.html),
+> which is deprecated and servers may be shut down permanently.
+> Users are encouraged to switch to
+> [intake-esgf](https://intake-esgf.readthedocs.io/), or use
+> [pystac-client](https://pystac-client.readthedocs.io/) to interact with the
+> new STAC-based search API. The new STAC servers are available at:
+> 
+> - https://discovery.east.esgf.io
+> - https://discovery.west.esgf.io
+>
+> Progress reports on the development of the new ESGF infrastructure can
+> be found on the [ESGF Roadmap GitHub
+> repository](https://github.com/ESGF/esgf-roadmap/blob/main/status/README.md).
+
+This package contains API code for calling the
+[legacy ESGF Search API](https://esgf.github.io/esg-search/ESGF_Search_RESTful_API.html)
+from Python code. It may be used to access data that is only findable
+through the legacy ESGF Search API, such as CMIP5, CORDEX, and older
+obs4MIPs data.
+
+You can try it online using Binder, or view the notebooks on NBViewer:
+[![Binder Launcher](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/ESGF/esgf-pyclient.git/master?filepath=notebooks)
+[![NBViewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.jupyter.org/github/ESGF/esgf-pyclient/tree/master/notebooks/)
+
+Please submit bugs and feature requests through the bug tracker on
+[GitHub](https://github.com/ESGF/esgf-pyclient). Pull requests are
+always welcome.
+
+Full [documentation](http://esgf-pyclient.readthedocs.org) is available
+on ReadTheDocs or in the docs directory.
+
+## Available servers
+
+At the time of writing (September 9, 2026), servers providing search
+using this API may be found at the following URLs:
+
+- <https://esgf-node.ornl.gov/esgf-1-5-bridge> (partially supports the legacy ESGF Search API, issues can be reported [here](https://github.com/esgf2-us/esg_fastapi/issues))
+- <https://esgf.ceda.ac.uk/esg-search/search>
+- <https://esgf-data.dkrz.de/esg-search/search>
+- <https://esg-dn1.nsc.liu.se/esg-search/search>
+
+while the following servers appear to be online, but return an error
+page:
+
+- <https://esgf-node.ipsl.upmc.fr/esg-search/search>
+- <https://esgf.nci.org.au/esg-search/search>
+- <https://esgf.nccs.nasa.gov/esg-search/search>
+- <https://esgdata.gfdl.noaa.gov/esg-search/search>
+
+Note that logging in using OpenID with the
+[`pyesgf.logon`](https://esgf-pyclient.readthedocs.io/en/latest/api.html#module-pyesgf-logon)
+module is no longer required to access ESGF data.
+All servers supporting this feature have been taken offline.

esgf_pyclient-0.3.2.dist-info/RECORD

@@ -0,0 +1,18 @@
+esgf_pyclient-0.3.2.dist-info/licenses/LICENSE,sha256=3iMAbo0g_5511YQjYqcO1AFcen0KUvm9PPCT9mIsm_c,1554
+pyesgf/__init__.py,sha256=bdeax8a2O8eHF1QvGk9c7Y6BTm-Xl6pAz6FUJNCj8wo,89
+pyesgf/exceptions.py,sha256=KtFtJFRK86Un8LnZlpzHl_n5uDeeUKQ86d18vHXCX-8,170
+pyesgf/logon.py,sha256=CoIDmY491Y6yUiCI0TfHC2rarAyXsFhP8IGMiDCPx1E,10667
+pyesgf/util.py,sha256=nR_i7VJRw6BtVMf_wKI1UVSlNEwbamwi88m_RWKqS98,4150
+pyesgf/search/__init__.py,sha256=QVAj8jIIvpApQii8XPU7MssG7010BlAS_Z_YokRirRo,574
+pyesgf/search/connection.py,sha256=PjsG0ABB2YK4e_Ve3ARO0ikLkdnO6uZ5i97BFNTNetU,13226
+pyesgf/search/constraints.py,sha256=F5r4SewiVT5ZUzV7wH10AiWfUi1tyiLgJWvzxe_XyRU,778
+pyesgf/search/consts.py,sha256=ocFMwTwNtwjRoD-qCUsfc3-cK24aFSXK-5KusGr3FK4,450
+pyesgf/search/context.py,sha256=ZQqVJNNICflOo2ozPfdV3UcYNLpLQhTpXWHAGeMIZ6k,13546
+pyesgf/search/exceptions.py,sha256=obhqko0Yy2f4dY-iuszyHmeBrG7_TjsIxoMKe2iruYk,183
+pyesgf/search/results.py,sha256=HvDQB7bPpcJgyDgPeGwFoBs_LP89dvTrNgLYE0wiVao,9618
+pyesgf/security/__init__.py,sha256=aBtm4Xxk_30gc8766VbAaVUrw1RprPl_smwhNjgOB64,379
+pyesgf/security/ats.py,sha256=PgrPNtybXREXoq6cgZzyWkwK_xYGqck6SvpXjOidzro,3195
+esgf_pyclient-0.3.2.dist-info/METADATA,sha256=Y-lknpSx3bhErZ5UuGlJfQFWkA7TIdfwNm0PSDDudTQ,5085
+esgf_pyclient-0.3.2.dist-info/WHEEL,sha256=YVMoNqKzERt-wjUZwJ33xBGAwnFl-4cqbYkTtWa4itE,91
+esgf_pyclient-0.3.2.dist-info/top_level.txt,sha256=XGYAbIglPZB5md4Z0gEUAQLkbU2zXrNUQLvvyakPjKo,7
+esgf_pyclient-0.3.2.dist-info/RECORD,,

esgf_pyclient-0.3.2.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (84.0.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

esgf_pyclient-0.3.2.dist-info/licenses/LICENSE

@@ -0,0 +1,29 @@
+BSD 3-Clause License
+
+Copyright (c) 2012, Science & Technology Facilities Council (STFC)
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

esgf_pyclient-0.3.2.dist-info/top_level.txt

@@ -0,0 +1 @@
+pyesgf

pyesgf/__init__.py

@@ -0,0 +1,6 @@
+"""
+Top-level package for Python interface to ESGF services.
+
+"""
+
+__version__ = '0.3.2'

pyesgf/exceptions.py

@@ -0,0 +1,16 @@
+"""
+Exceptions for the package go here
+
+"""
+
+
+class Error(Exception):
+    pass
+
+
+class OpenidResolutionError(Error):
+    pass
+
+
+class DuplicateHashError(Error):
+    pass

pyesgf/logon.py

@@ -0,0 +1,300 @@
+"""
+
+Module :mod:`pyesgf.logon`
+==========================
+
+Manage the client's interaction with ESGF's security system.  Using this
+module requires installing the MyProxyClient_ library.
+
+.. _MyProxyClient: http://pypi.python.org/pypi/MyProxyClient
+
+To obtain ESGF credentials create a :class:`LogonManager` instance and supply
+it with logon details::
+
+  >>> lm = LogonManager()
+  >>> lm.is_logged_on()
+  False
+  >>> lm.logon(username, password, myproxy_hostname, bootstrap=True)
+  >>> lm.is_logged_on()
+  True
+
+Logon parameters that aren't specified will be prompted for at the terminal
+by default.  The :class:`LogonManager` object also writes a ``.httprc`` file
+configuring OPeNDAP access through the NetCDF API.
+
+The option ``bootstrap=True`` is needed on the first run.
+
+You can use your OpenID to logon instead.  The logon details will be deduced
+from the OpenID where possible::
+
+  >>> lm.logoff()
+  >>> lm.is_logged_on()
+  False
+  >>> lm.logon_with_openid(openid, password, bootstrap=True)
+  >>> lm.is_logged_on()
+  True
+
+"""
+
+import os
+import os.path as op
+import shutil
+from defusedxml import ElementTree
+import requests
+import re
+from getpass import getpass
+
+try:
+    from myproxy.client import MyProxyClient
+    import OpenSSL
+    _has_myproxy = True
+except (ImportError, SyntaxError):
+    _has_myproxy = False
+
+from .exceptions import OpenidResolutionError
+
+# -----------------------------------------------------------------------------
+# Constants
+
+ESGF_DIR = op.join(os.path.expanduser('~'), '.esg')
+ESGF_CERTS_DIR = 'certificates'
+ESGF_CREDENTIALS = 'credentials.pem'
+DAP_CONFIG = op.join(os.path.expanduser('~'), '.dodsrc')
+DAP_CONFIG_MARKER = '<<< Managed by esgf-pyclient >>>'
+
+XRI_NS = 'xri://$xrd*($v*2.0)'
+MYPROXY_URN = 'urn:esg:security:myproxy-service'
+ESGF_OPENID_REXP = r'https://.*/esgf-idp/openid/(.*)'
+MYPROXY_URI_REXP = r'socket://([^:]*):?(\d+)?'
+
+# -----------------------------------------------------------------------------
+# classes
+
+
+class LogonManager(object):
+    """
+    Manages ESGF crendentials and security configuration files.
+
+    Also integrates with NetCDF's secure OPeNDAP configuration.
+
+    """
+    STATE_LOGGED_ON = 0
+    STATE_NO_CREDENTIALS = 1
+    STATE_EXPIRED_CREDENTIALS = 2
+    STATE_INVALID_CREDENTIALS = 3
+
+    def __init__(self, esgf_dir=ESGF_DIR, dap_config=DAP_CONFIG,
+                 verify=True):
+        """
+        :param esgf_dir: Root directory of ESGF state.  Default ~/.esg
+        :param dap_config: Set the location of .httprc.  Defaults to ~/.httprc
+        :param verify: SSL verification option. Default ``True``.
+
+        See the ``requests`` documenation to configure the
+        ``verify`` option:
+
+        http://docs.python-requests.org/en/master/user/advanced/#ssl-cert-verification
+
+        Note if dap_config is defined your current working directory must be
+        the same as the location as the dap_config file when OPeNDAP is
+        initialised.
+        """
+        if not _has_myproxy:
+            raise ImportError('pyesgf.logon requires MyProxyClient')
+
+        self.esgf_dir = esgf_dir
+        self.esgf_credentials = op.join(self.esgf_dir, ESGF_CREDENTIALS)
+        self.esgf_certs_dir = op.join(self.esgf_dir, ESGF_CERTS_DIR)
+        self.dap_config = dap_config
+        self.verify = verify
+
+        self._write_dap_config()
+
+    @property
+    def state(self):
+        if not op.exists(self.esgf_credentials):
+            return self.STATE_NO_CREDENTIALS
+        else:
+            with open(self.esgf_credentials) as fh:
+                data = fh.read()
+                cert = OpenSSL.crypto.load_certificate(
+                    OpenSSL.SSL.FILETYPE_PEM, data)
+
+            if cert.has_expired():
+                return self.STATE_EXPIRED_CREDENTIALS
+
+            # !TODO: check credentials against certificates
+
+            return self.STATE_LOGGED_ON
+
+    def is_logged_on(self):
+        return self.state == self.STATE_LOGGED_ON
+
+    def logon_with_openid(self, openid, password=None,
+                          bootstrap=False, update_trustroots=True,
+                          interactive=True):
+        """
+        Obtains ESGF credentials by detecting the MyProxy parameters from
+        the users OpenID.  Some ESGF compatible OpenIDs do not contain enough
+        information to obtain credentials.  In this case the user is prompted
+        for missing information if ``interactive == True``, otherwise an
+        exception is raised.
+
+        :param openid: OpenID to login with See :meth:`logon` for parameters
+            ``interactive``, ``bootstrap`` and ``update_trustroots``.
+        """
+        username, myproxy = self._get_logon_details(openid)
+        return self.logon(username, password, myproxy,
+                          bootstrap=bootstrap,
+                          update_trustroots=update_trustroots,
+                          interactive=interactive)
+
+    def logon(self, username=None, password=None, hostname=None,
+              bootstrap=False, update_trustroots=True,
+              interactive=True):
+        """
+        Obtain ESGF credentials from the specified MyProxy service.
+
+        If ``interactive == True`` then any missing parameters of ``password``,
+        ``username`` or ``hostname`` will be prompted for at the terminal.
+
+        :param interactive: Whether to ask for input at the terminal for
+            any missing information.  I.e. username, password or hostname.
+        :param bootstrap: Whether to bootstrap the trustroots for this
+            MyProxy service.
+        :param update_trustroots: Whether to update the trustroots for this
+            MyProxy service.
+
+        """
+        if interactive:
+            if hostname is None:
+                print('Enter myproxy hostname:', end=' ')
+                hostname = input()
+            if username is None:
+                print('Enter myproxy username:', end=' ')
+                username = input()
+            if password is None:
+                password = getpass('Enter password for %s: ' % username)
+
+        if None in (hostname, username, password):
+            raise OpenidResolutionError('Full logon details not available')
+
+        c = MyProxyClient(hostname=hostname, caCertDir=self.esgf_certs_dir)
+
+        creds = c.logon(username, password,
+                        bootstrap=bootstrap,
+                        updateTrustRoots=update_trustroots)
+        with open(self.esgf_credentials, 'wb') as fh:
+            for cred in creds:
+                fh.write(cred)
+
+    def logoff(self, clear_trustroots=False):
+        """
+        Remove any obtained credentials from the ESGF environment.
+
+        :param clear_trustroots: If True also remove trustroots.
+
+        """
+        if op.exists(self.esgf_credentials):
+            os.remove(self.esgf_credentials)
+
+        if clear_trustroots:
+            shutil.rmtree(self.esgf_certs_dir)
+
+    def _get_logon_details(self, openid):
+        response = requests.get(openid, verify=self.verify)
+        xml = ElementTree.fromstring(response.content)
+
+        hostname = None
+        port = None
+        username = None
+
+        services = xml.findall('.//{%s}Service' % XRI_NS)
+        for service in services:
+            try:
+                service_type = service.find('{%s}Type' % XRI_NS).text
+            except AttributeError:
+                continue
+
+            # Detect myproxy hostname and port
+            if service_type == MYPROXY_URN:
+                myproxy_uri = service.find('{%s}URI' % XRI_NS).text
+                mo = re.match(MYPROXY_URI_REXP, myproxy_uri)
+                if mo:
+                    hostname, port = mo.groups()
+
+        # If the OpenID matches the standard ESGF pattern assume it contains
+        # the username, otherwise prompt or raise an exception
+        mo = re.match(ESGF_OPENID_REXP, openid)
+        if mo:
+            username = mo.group(1)
+
+        # !TODO maybe support different myproxy port
+        if port is not None:
+            if not int(port) == 7512:
+                raise AssertionError()
+
+        return username, hostname
+
+    def _write_dap_config(self, verbose=False, validate=False):
+        preamble, managed, postamble = self._parse_dap_config()
+
+        with open(self.dap_config, 'w') as fh:
+            fh.write(('{preamble}\n'
+                      '# BEGIN {marker}\n'
+                      'HTTP.VERBOSE={verbose}\n'
+                      'HTTP.COOKIEJAR={esgf_dir}/.dods_cookies\n'
+                      'HTTP.SSL.VALIDATE=0\n'
+                      'HTTP.SSL.CERTIFICATE={esgf_dir}/credentials.pem\n'
+                      'HTTP.SSL.KEY={esgf_dir}/credentials.pem\n'
+                      'HTTP.SSL.CAPATH={esgf_certs_dir}\n'
+                      '# END {marker}\n'
+                      '{postamble}\n')
+                     .format(verbose=1 if verbose else 0,
+                             # validate=1 if validate else 0,
+                             esgf_certs_dir=self.esgf_certs_dir,
+                             esgf_dir=self.esgf_dir,
+                             marker=DAP_CONFIG_MARKER,
+                             preamble=preamble,
+                             postamble=postamble))
+
+    def _parse_dap_config(self, config_str=None):
+        """
+        Read the DAP_CONFIG file and extract the parts not controlled
+        by esgf-pyclient.
+
+        :return: (preamble, managed, postamble), three strings of
+            configuration lines before, within and after the esgf-pyclient
+            controlled block.
+
+        """
+        if config_str is None:
+            if not op.exists(self.dap_config):
+                return ('', '', '')
+            config_str = open(self.dap_config).read()
+
+        sections = re.split(r'^# (?:BEGIN|END) {0}$\n'
+                            .format(DAP_CONFIG_MARKER),
+                            config_str, flags=re.M)
+
+        if len(sections) < 2:
+            preamble, managed, postamble = sections[0], '', ''
+        elif len(sections) == 2:
+            preamble, managed, postamble = sections + ['']
+        elif len(sections) == 3:
+            preamble, managed, postamble = sections
+        else:
+            # In odd circumstances there might be more than 3 parts of the
+            # config so assume the final config is the one to keep
+            managed, unmanaged = [], []
+            sections.reverse()
+            while sections:
+                unmanaged.append(sections.pop())
+                if sections:
+                    managed.append(sections.pop())
+
+            preamble = '\n'.join(unmanaged[:-1])
+            postamble = unmanaged[-1]
+            managed = managed[-1]
+
+        return preamble.strip(), managed.strip(), postamble.strip()

pyesgf/search/__init__.py

@@ -0,0 +1,17 @@
+"""
+An interface to the `ESGF Search API`_
+
+
+.. _`ESGF Search API`: http://esgf.org/wiki/ESGF_Search_API
+
+"""
+
+from .connection import SearchConnection  # noqa: F401
+from .context import SearchContext  # noqa: F401
+from .constraints import GeospatialConstraint, any_of, not_equals  # noqa: F401
+from .results import ResultSet  # noqa: F401
+from .consts import TYPE_DATASET, TYPE_FILE  # noqa: F401
+
+# !TODO: ResultFormatter class.  process response json to specialise the result
+#        json.  Default is None
+# !TODO: pipe results to new process.  Command-line interface.