eodag 3.4.3__tar.gz → 3.5.1__tar.gz

{eodag-3.4.3 → eodag-3.5.1}/.github/workflows/deploy.yml

@@ -33,9 +33,12 @@ jobs:
         source .venv/bin/activate
         uv pip install python-semantic-release packaging
     - name: Set Git config
+      env:
+        TOKEN: ${{ secrets.USER_TOKEN }}
       run: |
         git config user.name "github-actions[bot]"
         git config user.email "'github-actions[bot]@users.noreply.github.com"
+        git remote set-url origin https://$TOKEN@github.com/${{ github.repository }}.git
     - name: Git merge develop in master
       run: |
         git checkout master
@@ -62,9 +65,12 @@ jobs:
         # Get history and tags for SCM versioning to work
         fetch-depth: 0
     - name: Set Git config
+      env:
+        TOKEN: ${{ secrets.USER_TOKEN }}
       run: |
         git config user.name "github-actions[bot]"
         git config user.email "'github-actions[bot]@users.noreply.github.com"
+        git remote set-url origin https://$TOKEN@github.com/${{ github.repository }}.git
     - name: Git merge master in develop
       run: |
         git checkout develop

{eodag-3.4.3 → eodag-3.5.1}/CHANGES.rst

@@ -3,6 +3,89 @@ Release history
 ===============
 
 
+v3.5.1 (2025-06-23)
+===================
+
+Bug Fixes
+---------
+
+* **core**: Enable count with search iterator (`#1700`_, `bbcc7ba`_)
+
+* **plugins**: Lru caching when fetching constraints with ECMWF (`#1698`_, `e23f47e`_)
+
+Refactoring
+-----------
+
+* **core**: Register downloader using manager from search to EOProduct (`#1699`_, `fd0c149`_)
+
+.. _#1698: https://github.com/CS-SI/eodag/pull/1698
+.. _#1699: https://github.com/CS-SI/eodag/pull/1699
+.. _#1700: https://github.com/CS-SI/eodag/pull/1700
+.. _bbcc7ba: https://github.com/CS-SI/eodag/commit/bbcc7ba311fcf25a0231203035166276e704ec8e
+.. _e23f47e: https://github.com/CS-SI/eodag/commit/e23f47ee97a50c0ba1d573801a17177c88f06eae
+.. _fd0c149: https://github.com/CS-SI/eodag/commit/fd0c149277735a3ecdc11588e8ac8e166b591ae8
+
+
+v3.5.0 (2025-06-20)
+===================
+
+Bug Fixes
+---------
+
+* **core**: Always validate PluginConfig before loading (`#1690`_, `59ac437`_)
+
+* **core**: Skip provider empty conf on init (`#1687`_, `0a4104e`_)
+
+* **plugins**: Raise errors when metadata discovery is not allowed (`#1534`_, `855ffa3`_)
+
+Build System
+------------
+
+* Update usgs to 0.3.6 (`#1688`_, `e63cfb1`_)
+
+Continuous Integration
+----------------------
+
+* Use personal access token for deploy github action (`#1693`_, `ff777d7`_)
+
+Documentation
+-------------
+
+* Dead-links and out-of-date param fix (`#1692`_, `445a20e`_)
+
+Features
+--------
+
+* **core**: Add env variable to whitelist providers (`#1672`_, `b93c4c8`_)
+
+* **core**: Add strict product types mode (`#1677`_, `5077fa5`_)
+
+* **plugins**: Auth token expiration margin (`#1665`_, `ef5fc18`_)
+
+* **server**: Added bbox filter support for collections search (`#1671`_, `5717f0d`_)
+
+.. _#1534: https://github.com/CS-SI/eodag/pull/1534
+.. _#1665: https://github.com/CS-SI/eodag/pull/1665
+.. _#1671: https://github.com/CS-SI/eodag/pull/1671
+.. _#1672: https://github.com/CS-SI/eodag/pull/1672
+.. _#1677: https://github.com/CS-SI/eodag/pull/1677
+.. _#1687: https://github.com/CS-SI/eodag/pull/1687
+.. _#1688: https://github.com/CS-SI/eodag/pull/1688
+.. _#1690: https://github.com/CS-SI/eodag/pull/1690
+.. _#1692: https://github.com/CS-SI/eodag/pull/1692
+.. _#1693: https://github.com/CS-SI/eodag/pull/1693
+.. _0a4104e: https://github.com/CS-SI/eodag/commit/0a4104e0518abc70e2133ca98472eea87d673a1c
+.. _445a20e: https://github.com/CS-SI/eodag/commit/445a20e060730642e703615c73225c0df3cc84d0
+.. _5077fa5: https://github.com/CS-SI/eodag/commit/5077fa591496811fb100c1e6b6a3e452cbdbe2a5
+.. _5717f0d: https://github.com/CS-SI/eodag/commit/5717f0deddbf022f2c6d5207ade77de6afb0f9d5
+.. _59ac437: https://github.com/CS-SI/eodag/commit/59ac437de01a8996d247b1f8239f332ed5dc5456
+.. _855ffa3: https://github.com/CS-SI/eodag/commit/855ffa39fa9b914eb39cc20d6e5c2cbbc1b2097a
+.. _b93c4c8: https://github.com/CS-SI/eodag/commit/b93c4c88f323af0eecb0950c90c6862ca9a7c3f4
+.. _e63cfb1: https://github.com/CS-SI/eodag/commit/e63cfb19ca64a2ed65f500ae9678e117a2ea4cf8
+.. _ef5fc18: https://github.com/CS-SI/eodag/commit/ef5fc188e515759c9227584b25805db75f537833
+.. _ff777d7: https://github.com/CS-SI/eodag/commit/ff777d7a1e33f612c5227dba4fecfcec55ff18fc
+
+
 v3.4.3 (2025-06-12)
 ===================
 

{eodag-3.4.3/eodag.egg-info → eodag-3.5.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eodag
-Version: 3.4.3
+Version: 3.5.1
 Summary: Earth Observation Data Access Gateway
 Home-page: https://github.com/CS-SI/eodag
 Author: CS GROUP - France
@@ -64,7 +64,7 @@ Requires-Dist: OWSLib>=0.27.1; extra == "csw"
 Provides-Extra: ecmwf
 Requires-Dist: ecmwf-api-client; extra == "ecmwf"
 Provides-Extra: usgs
-Requires-Dist: usgs>=0.3.1; extra == "usgs"
+Requires-Dist: usgs>=0.3.6; extra == "usgs"
 Provides-Extra: server
 Requires-Dist: fastapi>=0.93.0; extra == "server"
 Requires-Dist: pygeofilter; extra == "server"
@@ -317,7 +317,7 @@ An eodag instance can be exposed through a STAC compliant REST api from the comm
 
 .. code-block:: bash
 
-    docker run -p 5000:5000 --rm csspace/eodag-server:3.4.3
+    docker run -p 5000:5000 --rm csspace/eodag-server:3.5.1
 
 You can also browse over your STAC API server using `STAC Browser <https://github.com/radiantearth/stac-browser>`_.
 Simply run:

{eodag-3.4.3 → eodag-3.5.1}/README.rst

@@ -187,7 +187,7 @@ An eodag instance can be exposed through a STAC compliant REST api from the comm
 
 .. code-block:: bash
 
-    docker run -p 5000:5000 --rm csspace/eodag-server:3.4.3
+    docker run -p 5000:5000 --rm csspace/eodag-server:3.5.1
 
 You can also browse over your STAC API server using `STAC Browser <https://github.com/radiantearth/stac-browser>`_.
 Simply run:

{eodag-3.4.3 → eodag-3.5.1}/charts/eodag-server/Chart.yaml

@@ -1,5 +1,5 @@
 apiVersion: v2
-appVersion: 3.4.3
+appVersion: 3.5.1
 dependencies:
 - name: common
   repository:  oci://registry-1.docker.io/bitnamicharts
@@ -15,4 +15,4 @@ name: eodag-server
 sources:
 - https://github.com/CS-SI/eodag
 type: application
-version: 3.4.3
+version: 3.5.1

{eodag-3.4.3 → eodag-3.5.1}/docs/getting_started_guide/configure.rst

@@ -157,6 +157,26 @@ Some EODAG core settings can be overriden using environment variables:
   <https://eodag.readthedocs.io/en/stable/notebooks/api_user_guide/2_providers_products_available.html#Product-types-discovery>`_
   in place of https://cs-si.github.io/eodag/eodag/resources/ext_product_types.json.
   If the file is not readable, only user-modified providers will be fetched.
+* ``EODAG_PROVIDERS_WHITELIST`` to restrict EODAG to only use a specific list of providers.
+
+  If this environment variable is set (as a comma-separated list of provider names), EODAG will only load and use the specified providers.
+  All other providers will be ignored, regardless of their presence in configuration files.
+
+  This is useful for restricting EODAG to a subset of providers, for example in controlled or production environments.
+* ``EODAG_STRICT_PRODUCT_TYPES`` to control how product types are listed.
+
+  If this environment variable is set to a truthy value (such as ``1``, ``true``, ``yes``, or ``on``), EODAG will only list product types that are present in the main product types configuration file.
+  Product types defined only in provider configurations (but not in the main product types configuration) will be ignored.
+  If not set, EODAG will also include product types defined only in provider configurations, with minimal metadata.
+
+  This is useful if you want to strictly control which product types are available, for example to ensure consistency across environments.
+
+Example usage:
+
+.. code-block:: bash
+
+   export EODAG_PROVIDERS_WHITELIST=peps,creodias,theia
+   export EODAG_STRICT_PRODUCT_TYPES=true
 
 CLI configuration
 ^^^^^^^^^^^^^^^^^

{eodag-3.4.3 → eodag-3.5.1}/docs/getting_started_guide/register.rst

@@ -150,7 +150,7 @@ No authentication needed.
 
 ``ecmwf``
 ^^^^^^^^^
-Create an account `here <https://apps.ecmwf.int/registration/>`__.
+Create an account `here <https://www.ecmwf.int/user/login>`__.
 
 Then use *email* as ``username`` and *key* as ``password`` from `here <https://api.ecmwf.int/v1/key/>`__ in eodag credentials.
 EODAG can be used to request for public datasets as for operational archive. Please note that for public datasets you

{eodag-3.4.3 → eodag-3.5.1}/docs/stac_rest.rst

@@ -115,7 +115,7 @@ available on `https://hub.docker.com/r/csspace/eodag-server <https://hub.docker.
 
 .. code-block:: bash
 
-    $ docker run -p 5000:5000 --rm csspace/eodag-server:3.4.3
+    $ docker run -p 5000:5000 --rm csspace/eodag-server:3.5.1
 
 Example
 -------

{eodag-3.4.3 → eodag-3.5.1}/eodag/api/core.py

@@ -36,7 +36,7 @@ from whoosh.index import exists_in, open_dir
 from whoosh.qparser import QueryParser
 
 from eodag.api.product.metadata_mapping import (
-    ONLINE_STATUS,
+    NOT_AVAILABLE,
     mtd_cfg_as_conversion_and_querypath,
 )
 from eodag.api.search_result import SearchResult
@@ -79,6 +79,7 @@ from eodag.utils import (
     string_to_jsonpath,
     uri_to_path,
 )
+from eodag.utils.env import is_env_var_true
 from eodag.utils.exceptions import (
     AuthenticationError,
     EodagError,
@@ -175,12 +176,19 @@ class EODataAccessGateway:
         share_credentials(self.providers_config)
 
         # init updated providers conf
+        strict_mode = is_env_var_true("EODAG_STRICT_PRODUCT_TYPES")
+        available_product_types = set(self.product_types_config.source.keys())
+
         for provider in self.providers_config.keys():
             provider_config_init(
                 self.providers_config[provider],
                 load_stac_provider_config(),
             )
 
+            self._sync_provider_product_types(
+                provider, available_product_types, strict_mode
+            )
+
         # re-build _plugins_manager using up-to-date providers_config
         self._plugins_manager.rebuild(self.providers_config)
 
@@ -226,6 +234,61 @@ class EODataAccessGateway:
                     )
         self.set_locations_conf(locations_conf_path)
 
+    def _sync_provider_product_types(
+        self,
+        provider: str,
+        available_product_types: set[str],
+        strict_mode: bool,
+    ) -> None:
+        """
+        Synchronize product types for a provider based on strict or permissive mode.
+
+        In strict mode, removes product types not in available_product_types.
+        In permissive mode, adds empty product type configs for missing types.
+
+        :param provider: The provider name whose product types should be synchronized.
+        :param available_product_types: The set of available product type IDs.
+        :param strict_mode: If True, remove unknown product types; if False, add empty configs for them.
+        :returns: None
+        """
+        provider_products = self.providers_config[provider].products
+        products_to_remove: list[str] = []
+        products_to_add: list[str] = []
+
+        for product_id in provider_products:
+            if product_id == GENERIC_PRODUCT_TYPE:
+                continue
+
+            if product_id not in available_product_types:
+                if strict_mode:
+                    products_to_remove.append(product_id)
+                    continue
+
+                empty_product = {
+                    "title": product_id,
+                    "abstract": NOT_AVAILABLE,
+                }
+                self.product_types_config.source[
+                    product_id
+                ] = empty_product  # will update available_product_types
+                products_to_add.append(product_id)
+
+        if products_to_add:
+            logger.debug(
+                "Product types permissive mode, %s added (provider %s)",
+                ", ".join(products_to_add),
+                provider,
+            )
+
+        if products_to_remove:
+            logger.debug(
+                "Product types strict mode, ignoring %s (provider %s)",
+                ", ".join(products_to_remove),
+                provider,
+            )
+            for id in products_to_remove:
+                del self.providers_config[provider].products[id]
+
     def get_version(self) -> str:
         """Get eodag package version"""
         return version("eodag")
@@ -608,11 +671,14 @@ class EODataAccessGateway:
             for product_type_id in p.products:  # type: ignore
                 if product_type_id == GENERIC_PRODUCT_TYPE:
                     continue
+
                 config = self.product_types_config[product_type_id]
                 config["_id"] = product_type_id
+
                 if "alias" in config:
                     product_type_id = config["alias"]
                 product_type = {"ID": product_type_id, **config}
+
                 if product_type not in product_types:
                     product_types.append(product_type)
 
@@ -620,11 +686,18 @@ class EODataAccessGateway:
         return sorted(product_types, key=itemgetter("ID"))
 
     def fetch_product_types_list(self, provider: Optional[str] = None) -> None:
-        """Fetch product types list and update if needed
+        """Fetch product types list and update if needed.
+
+        If strict mode is enabled (by setting the ``EODAG_STRICT_PRODUCT_TYPES`` environment variable
+        to a truthy value), this method will not fetch or update product types and will return immediately.
 
         :param provider: The name of a provider or provider-group for which product types
                          list should be updated. Defaults to all providers (None value).
         """
+        strict_mode = is_env_var_true("EODAG_STRICT_PRODUCT_TYPES")
+        if strict_mode:
+            return
+
         providers_to_fetch = list(self.providers_config.keys())
         # check if some providers are grouped under a group name which is not a provider name
         if provider is not None and provider not in self.providers_config:
@@ -1321,7 +1394,11 @@ class EODataAccessGateway:
         prev_product = None
         next_page_url = None
         next_page_query_obj = None
+        number_matched = None
         while True:
+            # if count is enabled, it will only be performed on 1st iteration
+            if iteration == 2:
+                kwargs["count"] = False
             if iteration > 1 and next_page_url:
                 pagination_config["next_page_url_tpl"] = next_page_url
             if iteration > 1 and next_page_query_obj:
@@ -1329,11 +1406,13 @@ class EODataAccessGateway:
             logger.info("Iterate search over multiple pages: page #%s", iteration)
             try:
                 # remove unwanted kwargs for _do_search
-                kwargs.pop("count", None)
                 kwargs.pop("raise_errors", None)
                 search_result = self._do_search(
-                    search_plugin, count=False, raise_errors=True, **kwargs
+                    search_plugin, raise_errors=True, **kwargs
                 )
+                # if count is enabled, it will only be performed on 1st iteration
+                if iteration == 1:
+                    number_matched = search_result.number_matched
             except Exception:
                 logger.warning(
                     "error at retrieval of data from %s, for params: %s",
@@ -1388,6 +1467,8 @@ class EODataAccessGateway:
                     )
                     last_page_with_products = iteration - 1
                     break
+                # use count got from 1st iteration
+                search_result.number_matched = number_matched
                 yield search_result
                 prev_product = product
                 # Prevent a last search if the current one returned less than the
@@ -1475,6 +1556,9 @@ class EODataAccessGateway:
                 )
                 self.fetch_product_types_list()
 
+        # remove unwanted count
+        kwargs.pop("count", None)
+
         search_plugins, search_kwargs = self._prepare_search(
             start=start, end=end, geom=geom, locations=locations, **kwargs
         )
@@ -1497,6 +1581,7 @@ class EODataAccessGateway:
                 for page_results in self.search_iter_page_plugin(
                     items_per_page=itp,
                     search_plugin=search_plugin,
+                    count=False,
                     **search_kwargs,
                 ):
                     all_results.data.extend(page_results.data)
@@ -1907,29 +1992,7 @@ class EODataAccessGateway:
                     logger.debug("product type %s not found", eo_product.product_type)
 
                 if eo_product.search_intersection is not None:
-                    download_plugin = self._plugins_manager.get_download_plugin(
-                        eo_product
-                    )
-                    if len(eo_product.assets) > 0:
-                        matching_url = next(iter(eo_product.assets.values()))["href"]
-                    elif eo_product.properties.get("storageStatus") != ONLINE_STATUS:
-                        matching_url = eo_product.properties.get(
-                            "orderLink"
-                        ) or eo_product.properties.get("downloadLink")
-                    else:
-                        matching_url = eo_product.properties.get("downloadLink")
-
-                    try:
-                        auth_plugin = next(
-                            self._plugins_manager.get_auth_plugins(
-                                search_plugin.provider,
-                                matching_url=matching_url,
-                                matching_conf=download_plugin.config,
-                            )
-                        )
-                    except StopIteration:
-                        auth_plugin = None
-                    eo_product.register_downloader(download_plugin, auth_plugin)
+                    eo_product._register_downloader_from_manager(self._plugins_manager)
 
             results.extend(res)
             total_results = (

{eodag-3.4.3 → eodag-3.5.1}/eodag/api/product/_product.py

@@ -43,6 +43,7 @@ from eodag.api.product.metadata_mapping import (
     DEFAULT_GEOMETRY,
     NOT_AVAILABLE,
     NOT_MAPPED,
+    ONLINE_STATUS,
 )
 from eodag.utils import (
     DEFAULT_DOWNLOAD_TIMEOUT,
@@ -62,6 +63,7 @@ if TYPE_CHECKING:
     from eodag.plugins.apis.base import Api
     from eodag.plugins.authentication.base import Authentication
     from eodag.plugins.download.base import Download
+    from eodag.plugins.manager import PluginManager
     from eodag.types.download_args import DownloadConf
     from eodag.utils import Unpack
 
@@ -238,6 +240,37 @@ class EOProduct:
                 f"Unable to get {e.args[0]} key from EOProduct.properties"
             )
 
+    def _register_downloader_from_manager(self, plugins_manager: PluginManager) -> None:
+        """Register the downloader and authenticator for this EOProduct using the
+        provided plugins manager.
+        This method is typically called after the EOProduct has been created and
+        before any download operation is performed.
+
+        :param plugins_manager: The plugins manager instance to use for retrieving
+                                the download and authentication plugins.
+        """
+        download_plugin = plugins_manager.get_download_plugin(self)
+        if len(self.assets) > 0:
+            matching_url = next(iter(self.assets.values()))["href"]
+        elif self.properties.get("storageStatus") != ONLINE_STATUS:
+            matching_url = self.properties.get("orderLink") or self.properties.get(
+                "downloadLink"
+            )
+        else:
+            matching_url = self.properties.get("downloadLink")
+
+        try:
+            auth_plugin = next(
+                plugins_manager.get_auth_plugins(
+                    self.provider,
+                    matching_url=matching_url,
+                    matching_conf=download_plugin.config,
+                )
+            )
+        except StopIteration:
+            auth_plugin = None
+        self.register_downloader(download_plugin, auth_plugin)
+
     def register_downloader(
         self, downloader: Union[Api, Download], authenticator: Optional[Authentication]
     ) -> None:

{eodag-3.4.3 → eodag-3.5.1}/eodag/api/product/metadata_mapping.py

@@ -54,6 +54,7 @@ from eodag.utils import (
     string_to_jsonpath,
     update_nested_dict,
 )
+from eodag.utils.exceptions import ValidationError
 
 if TYPE_CHECKING:
     from shapely.geometry.base import BaseGeometry
@@ -1286,7 +1287,10 @@ def mtd_cfg_as_conversion_and_querypath(
 
 
 def format_query_params(
-    product_type: str, config: PluginConfig, query_dict: dict[str, Any]
+    product_type: str,
+    config: PluginConfig,
+    query_dict: dict[str, Any],
+    error_context: str = "",
 ) -> dict[str, Any]:
     """format the search parameters to query parameters"""
     if "raise_errors" in query_dict.keys():
@@ -1299,10 +1303,26 @@ def format_query_params(
         **config.products.get(product_type, {}).get("metadata_mapping", {}),
     )
 
+    # Raise error if non-queryables parameters are used and raise_mtd_discovery_error configured
+    if (
+        raise_mtd_discovery_error := config.products.get(product_type, {})
+        .get("discover_metadata", {})
+        .get("raise_mtd_discovery_error")
+    ) is None:
+        raise_mtd_discovery_error = getattr(config, "discover_metadata", {}).get(
+            "raise_mtd_discovery_error", False
+        )
+
     query_params: dict[str, Any] = {}
     # Get all the search parameters that are recognised as queryables by the
     # provider (they appear in the queryables dictionary)
-    queryables = _get_queryables(query_dict, config, product_type_metadata_mapping)
+    queryables = _get_queryables(
+        query_dict,
+        config,
+        product_type_metadata_mapping,
+        raise_mtd_discovery_error,
+        error_context,
+    )
 
     for eodag_search_key, provider_search_key in queryables.items():
         user_input = query_dict[eodag_search_key]
@@ -1438,6 +1458,8 @@ def _get_queryables(
     search_params: dict[str, Any],
     config: PluginConfig,
     metadata_mapping: dict[str, Any],
+    raise_mtd_discovery_error: bool,
+    error_context: str,
 ) -> dict[str, Any]:
     """Retrieve the metadata mappings that are query-able"""
     logger.debug("Retrieving queryable metadata from metadata_mapping")
@@ -1445,6 +1467,13 @@ def _get_queryables(
     for eodag_search_key, user_input in search_params.items():
         if user_input is not None:
             md_mapping = metadata_mapping.get(eodag_search_key, (None, NOT_MAPPED))
+            # raise an error when a query param not allowed by the provider is found
+            if not isinstance(md_mapping, list) and raise_mtd_discovery_error:
+                raise ValidationError(
+                    "Search parameters which are not queryable are disallowed for this product type on this provider: "
+                    f"please remove '{eodag_search_key}' from your search parameters. {error_context}",
+                    {eodag_search_key},
+                )
             _, md_value = md_mapping
             # query param from defined metadata_mapping
             if md_mapping is not None and isinstance(md_mapping, list):