hdx-python-api 6.7.2__tar.gz → 6.7.3__tar.gz

{hdx_python_api-6.7.2 → hdx_python_api-6.7.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: hdx-python-api
-Version: 6.7.2
+Version: 6.7.3
 Summary: HDX Python API for interacting with the Humanitarian Data Exchange
 Project-URL: Homepage, https://github.com/OCHA-DAP/hdx-python-api
 Author: Michael Rans

{hdx_python_api-6.7.2 → hdx_python_api-6.7.3}/documentation/index.md

@@ -36,6 +36,7 @@ upload your datasets to HDX.
     -   [Location Management](#location-management)
     -   [Vocabulary Management](#vocabulary-management)
     -   [Activity Streams](#activity-streams)
+    -   [HDX Statistics](#hdx-statistics)
     -   [Pipeline State](#pipeline-state)
 -   [Working Examples](#working-examples)
 -   [Project Framework](#project-framework)
@@ -636,6 +637,16 @@ function, for example:
 
     dataset.remove_showcase(showcase)
 
+HDX runs file structure checks on resources, recording schema changes (eg.
+column names, order or types) and whether sheets have merged cells. You can get
+these checks for all of a dataset's resources from HDX:
+
+    fs_check_info = dataset.get_fs_check_info()
+
+This returns a list of dictionaries, each from a resource name to a list of
+check events with keys such as `state`, `timestamp`, `sheet_changes` and
+`hxl_proxy_response`.
+
 ### Time Period
 
 Time Period is a mandatory field in HDX. It is the earliest start date and
@@ -947,6 +958,12 @@ If the method **set_file_to_upload** is used to supply a file, the resource
 `last_modified` field is set to now automatically regardless of the value of
 `data_updated` or whether **mark_data_updated** has been called.
 
+The file structure check events of a resource (see Dataset Specific
+Operations) can be obtained from its `fs_check_info` field without a call to
+HDX:
+
+    events = resource.get_fs_check_info()
+
 ### Data Dictionary
 
 A resource can have a data dictionary describing the columns in the data it
@@ -974,11 +991,36 @@ using the getter:
 
     data_dictionary = resource.get_hdx_data_dictionary()
 
+To change the data dictionary of a resource already in HDX without sending
+dataset update notifications, use **update_hdx_data_dictionary**. It patches
+only the `hdx_data_dictionary` field (so `last_modified` is unchanged), then,
+if the resource has a datastore, updates the datastore field labels and
+descriptions without reloading the data. The given columns are merged by
+`field` into the existing data dictionary in the resource metadata, so read
+the resource from HDX first. Pass `replace=True` to replace the data
+dictionary instead and `update_datastore=False` to leave the datastore alone:
+
+    resource = Resource.read_from_hdx(RESOURCE_ID)
+    resource.update_hdx_data_dictionary([
+        {
+            "field": "col_name",
+            "label": "Column Label",
+            "description": "New description.",
+        },
+    ])
+
+Columns are matched to datastore fields by `field`. A warning is logged for
+any column with no matching datastore field.
+
 ### Datastore
 
-A resource's data can be loaded into the HDX datastore and queried. HDX
-requires an API key for all datastore actions and dumps, including reads of
-public resources. To create, update or delete the datastore for a resource:
+A resource's data can be loaded into the HDX datastore and queried. HDX calls this
+[Tabular Data Endpoints (TDE)](https://docs.humdata.org/build/hdx-apis/tabular-data-endpoints),
+and it is available only for resources with TDE enabled (their `datastore_active`
+field is true). Contact hdx@un.org to request TDE for your datasets. HDX requires an
+API key for all datastore actions and dumps, including reads of public resources,
+and limits each IP address to 60 requests per minute. To create, update or delete
+the datastore for a resource:
 
     resource.create_datastore(
         [{"id": "code", "type": "text"}, {"id": "value", "type": "numeric"}],
@@ -988,6 +1030,17 @@ public resources. To create, update or delete the datastore for a resource:
     has_datastore = resource.has_datastore()
     resource.delete_datastore()
 
+HDX loads uploaded files into the datastore using DataPusher+. To reload a
+resource's file into the datastore without uploading it again (so
+`last_modified` is unchanged), use **submit_to_datapusher**. It returns False
+if the job was not submitted, eg. because one is already pending. HDX's
+datastore allowlist and format checks, which apply to automatic submissions
+on upload, are not applied. **get_datapusher_status** returns the status of
+the latest job or None if there is none:
+
+    submitted = resource.submit_to_datapusher(ignore_hash=True)
+    status = resource.get_datapusher_status()  # eg. status["status"] == "complete"
+
 To download the whole datastore (rather than the resource's file) use
 **download_datastore**, which returns the URL and path like **download**.
 `file_format` can be `csv` (default), `tsv`, `json` or `xml`, and `filename`
@@ -1008,7 +1061,8 @@ and `include_total`, and passes any other keyword arguments to CKAN:
     records = result["records"]
 
 **get_datastore_records** takes the same query parameters but pages through
-all matching rows, yielding each record (`page_size` defaults to 10000):
+all matching rows, yielding each record (`page_size` defaults to 10000).
+`records_format` csv and tsv are not supported; use `search_datastore` for those:
 
     for record in resource.get_datastore_records(filters={"code": "A"}):
         ...
@@ -1020,7 +1074,9 @@ internal `_id` field:
 
 **search_datastore_sql** wraps CKAN's `datastore_search_sql`, running a
 SELECT statement where tables are named by double quoted resource id and
-returning the list of records:
+returning the list of records. HDX returns at most 32,000 rows per query and
+cuts off larger results, in which case a warning is logged (see
+[paging in the TDE docs](https://docs.humdata.org/build/hdx-apis/tabular-data-endpoints/troubleshooting-and-error-handling#pagination)):
 
     records = Resource.search_datastore_sql(
         'SELECT code, SUM(value) AS total FROM "RESOURCE_ID" GROUP BY code'
@@ -1067,6 +1123,11 @@ configuration):
 
     user = User.get_current_user()
 
+You can get the name of the API token in the configuration and the md5 hash
+of its owner's email:
+
+    token_info = User.get_current_token_info()  # {"email_hash": ..., "token_name": ...}
+
 You can check that the current user has a particular permission to a specific
 organization:
 
@@ -1223,6 +1284,20 @@ permission to edit the object the activity relates to for these:
     data = activity.get_data()
     diff = activity.get_diff(diff_type="unified")  # or "context" or "html"
 
+## HDX Statistics
+
+General statistics about datasets, organizations and users on HDX can be
+obtained as follows:
+
+    from hdx.api.utilities.hdx_stats import get_general_statistics, push_general_stats
+
+    stats = get_general_statistics()
+
+Users with permission to manage basic scheduled tasks can compute the statistics
+and push them to analytics:
+
+    pushed_stats = push_general_stats()
+
 ## Pipeline State
 
 The HDXState class allows the reading and writing of state to a given dataset. Input

{hdx_python_api-6.7.2 → hdx_python_api-6.7.3}/src/hdx/api/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '6.7.2'
-__version_tuple__ = version_tuple = (6, 7, 2)
+__version__ = version = '6.7.3'
+__version_tuple__ = version_tuple = (6, 7, 3)
 
 __commit_id__ = commit_id = None

hdx_python_api-6.7.3/src/hdx/api/utilities/hdx_stats.py

@@ -0,0 +1,47 @@
+"""HDX site-wide statistics"""
+
+from hdx.api.configuration import Configuration
+from hdx.data.hdxobject import HDXError
+
+
+def _call_stats_action(action: str, configuration: Configuration | None) -> dict:
+    """Call an HDX statistics action
+
+    Args:
+        action: CKAN action name
+        configuration: HDX configuration. Defaults to global configuration.
+
+    Returns:
+        Statistics dictionary
+    """
+    if configuration is None:
+        configuration = Configuration.read()
+    try:
+        return configuration.call_remoteckan(action, {})
+    except Exception as e:
+        raise HDXError(f"Failed when trying to call {action}!") from e
+
+
+def get_general_statistics(configuration: Configuration | None = None) -> dict:
+    """Get general statistics about datasets, organizations and users on HDX
+
+    Args:
+        configuration: HDX configuration. Defaults to global configuration.
+
+    Returns:
+        Dictionary with keys datasets, organizations and users
+    """
+    return _call_stats_action("hdx_general_statistics", configuration)
+
+
+def push_general_stats(configuration: Configuration | None = None) -> dict:
+    """Compute general statistics on HDX and push them to analytics. Requires
+    permission to manage basic scheduled tasks.
+
+    Args:
+        configuration: HDX configuration. Defaults to global configuration.
+
+    Returns:
+        Statistics that were pushed
+    """
+    return _call_stats_action("hdx_push_general_stats", configuration)

{hdx_python_api-6.7.2 → hdx_python_api-6.7.3}/src/hdx/data/activity.py

@@ -129,7 +129,9 @@ class Activity(HDXObject):
             result = activity._write_to_hdx(action, data)
             return [cls(x, configuration=configuration) for x in result]
         data = dict(data)
-        limit = data.setdefault("limit", 100)
+        # HDX silently caps limit at ckan.activity_list_limit_max
+        limit = min(data.get("limit", 100), 100)
+        data["limit"] = limit
         offset = data.get("offset", 0)
         activities = []
         while True:

{hdx_python_api-6.7.2 → hdx_python_api-6.7.3}/src/hdx/data/dataset.py

@@ -147,6 +147,7 @@ class Dataset(HDXObject):
             "list": "package_list",
             "autocomplete": "package_autocomplete",
             "create_default_views": "package_create_default_resource_views",
+            "fs_check_info": "fs_check_info_show",
         }
 
     def __setitem__(self, key: Any, value: Any) -> None:
@@ -1447,13 +1448,7 @@ class Dataset(HDXObject):
         Returns:
             List of Activity objects
         """
-        return Activity.get_activities(
-            "package",
-            self.data.get("id") or self.data.get("name"),
-            self.configuration,
-            all_pages,
-            **kwargs,
-        )
+        return self._get_activities("package", all_pages, **kwargs)
 
     @classmethod
     def autocomplete(
@@ -2333,6 +2328,25 @@ class Dataset(HDXObject):
             return None
         return f"{self.configuration.get_hdx_site_url()}/api/3/action/package_show?id={name_or_id}"
 
+    def get_fs_check_info(self) -> list[dict]:
+        """Get file structure check information for the dataset's resources
+        from HDX. Each list item is a dictionary from resource name to a list
+        of check events, each of which can have keys such as state, timestamp,
+        sheet_changes and hxl_proxy_response.
+
+        Returns:
+            List of dictionaries from resource name to file structure checks
+        """
+        success, result = self._read_from_hdx(
+            "dataset",
+            self.get_name_or_id(prefer_name=False),
+            "id",
+            self.actions()["fs_check_info"],
+        )
+        if success:
+            return result
+        return []
+
     def generate_resource(
         self,
         folder: Path | str,

{hdx_python_api-6.7.2 → hdx_python_api-6.7.3}/src/hdx/data/hdxobject.py

@@ -9,7 +9,7 @@ from collections import UserDict
 from collections.abc import Sequence
 from os.path import isfile
 from pathlib import Path
-from typing import Any, Optional, Union
+from typing import TYPE_CHECKING, Any, Optional, Union
 
 from ckanapi.errors import NotFound
 from hdx.utilities.dictandlist import merge_two_dictionaries
@@ -20,6 +20,9 @@ from hdx.utilities.loader import (
 
 from hdx.api.configuration import Configuration
 
+if TYPE_CHECKING:
+    from hdx.data.activity import Activity
+
 logger = logging.getLogger(__name__)
 
 
@@ -376,6 +379,30 @@ class HDXObject(UserDict, ABC):
             **kwargs,
         )
 
+    def _get_activities(
+        self, object_type: str, all_pages: bool = False, **kwargs: Any
+    ) -> list["Activity"]:
+        """Helper method to get the activity stream of this object
+
+        Args:
+            object_type: One of package, organization, group or user
+            all_pages: Whether to get all pages of results. Defaults to False.
+            **kwargs: Parameters to pass to Activity.get_activities
+
+        Returns:
+            List of Activity objects
+        """
+        # Imported here as hdx.data.activity imports this module
+        from hdx.data.activity import Activity
+
+        return Activity.get_activities(
+            object_type,
+            self.data.get("id") or self.data.get("name"),
+            self.configuration,
+            all_pages,
+            **kwargs,
+        )
+
     def _write_to_hdx(
         self,
         action: str,

{hdx_python_api-6.7.2 → hdx_python_api-6.7.3}/src/hdx/data/location.py

@@ -232,13 +232,7 @@ class Location(HDXObject):
         Returns:
             List of Activity objects
         """
-        return Activity.get_activities(
-            "group",
-            self.data.get("id") or self.data.get("name"),
-            self.configuration,
-            all_pages,
-            **kwargs,
-        )
+        return self._get_activities("group", all_pages, **kwargs)
 
     @classmethod
     def autocomplete(

{hdx_python_api-6.7.2 → hdx_python_api-6.7.3}/src/hdx/data/organization.py

@@ -326,13 +326,7 @@ class Organization(HDXObject):
         Returns:
             List of Activity objects
         """
-        return Activity.get_activities(
-            "organization",
-            self.data.get("id") or self.data.get("name"),
-            self.configuration,
-            all_pages,
-            **kwargs,
-        )
+        return self._get_activities("organization", all_pages, **kwargs)
 
     @classmethod
     def autocomplete(

{hdx_python_api-6.7.2 → hdx_python_api-6.7.3}/src/hdx/data/resource.py

@@ -72,6 +72,8 @@ class Resource(HDXObject):
             "datastore_upsert": "datastore_upsert",
             "datastore_search": "datastore_search",
             "datastore_search_sql": "datastore_search_sql",
+            "datapusher_submit": "datapusher_submit",
+            "datapusher_status": "datapusher_status",
         }
 
     def update_from_yaml(
@@ -319,6 +321,69 @@ class Resource(HDXObject):
             data_dictionary, separators=(",", ":")
         )
 
+    def update_hdx_data_dictionary(
+        self,
+        data_dictionary: Sequence[dict],
+        replace: bool = False,
+        update_datastore: bool = True,
+    ) -> None:
+        """Update the resource's data dictionary (column definitions) in HDX
+        without changing last_modified, so no dataset update notifications are
+        sent. Column definitions are as for set_hdx_data_dictionary. Unless
+        replace is True, they are merged by field into the existing data
+        dictionary in the resource metadata (so call read_from_hdx first).
+        The resource's hdx_data_dictionary field is patched, then if
+        update_datastore is True and the resource has a datastore, the
+        datastore field labels and descriptions are updated using
+        datastore_create without reloading the data.
+
+        Args:
+            data_dictionary: Sequence of column definition dictionaries
+            replace: Replace rather than merge into existing data dictionary. Defaults to False.
+            update_datastore: Update datastore field labels and descriptions. Defaults to True.
+
+        Returns:
+            None
+        """
+        if replace:
+            merged = [dict(column) for column in data_dictionary]
+        else:
+            new_columns = {column["field"]: column for column in data_dictionary}
+            merged = []
+            for column in self.get_hdx_data_dictionary() or []:
+                new_column = new_columns.pop(column.get("field"), None)
+                merged.append({**column, **new_column} if new_column else column)
+            merged.extend(new_columns.values())
+        self.set_hdx_data_dictionary(merged)
+        self._write_to_hdx(
+            "patch",
+            {
+                "id": self.data["id"],
+                "hdx_data_dictionary": self.data["hdx_data_dictionary"],
+            },
+            "id",
+        )
+        if not update_datastore or not self.has_datastore():
+            return
+        columns = {column["field"]: column for column in merged}
+        fields = []
+        for field in self.get_datastore_fields():
+            info = dict(field.get("info") or {})
+            column = columns.pop(field["id"], None)
+            if column:
+                info["label"] = column["label"]
+                info["notes"] = column["description"]
+            fields.append({"id": field["id"], "type": field["type"], "info": info})
+        if columns:
+            logger.warning(
+                f"Data dictionary fields {', '.join(columns)} not in datastore for resource {self.data['id']}!"
+            )
+        self._write_to_hdx(
+            "datastore_create",
+            {"resource_id": self.data["id"], "force": True, "fields": fields},
+            "resource_id",
+        )
+
     def clean_format(self) -> str:
         """Clean the resource's format, setting it to None if it is invalid and
         cannot be mapped
@@ -683,6 +748,30 @@ class Resource(HDXObject):
             logger.debug(result)
         return resources
 
+    def _download_file(
+        self,
+        url: str,
+        folder: Path | str | None,
+        filename: str,
+        retriever: Retrieve | None,
+    ) -> Path:
+        """Helper method to download a file from HDX
+
+        Args:
+            url: URL to download
+            folder: Folder to download file to. None means temporary folder.
+            filename: Filename to use
+            retriever: Retrieve object to use. None means do not use one.
+
+        Returns:
+            Path to downloaded file
+        """
+        session = hdx.api.utilities.url_utils.get_ckan_ready_session(self.configuration)
+        with Download(session=session) as downloader:
+            if retriever:
+                downloader = retriever.clone(downloader)
+            return downloader.download_file(url, folder=folder, filename=filename)
+
     def download(
         self, folder: Path | str | None = None, retriever: Retrieve | None = None
     ) -> tuple[str, Path]:
@@ -706,12 +795,7 @@ class Resource(HDXObject):
         file_format = f".{self.get_format()}"
         if not filename.endswith(file_format):
             filename = f"{filename}{file_format}"
-        session = hdx.api.utilities.url_utils.get_ckan_ready_session(self.configuration)
-        with Download(session=session) as downloader:
-            if retriever:
-                downloader = retriever.clone(downloader)
-            path = downloader.download_file(url, folder=folder, filename=filename)
-            return url, path
+        return url, self._download_file(url, folder, filename, retriever)
 
     def get_datastore_dump_url(self, file_format: str = "csv") -> str:
         """Get the URL from which the resource datastore can be downloaded in
@@ -755,19 +839,15 @@ class Resource(HDXObject):
         Returns:
             (URL downloaded, Path to downloaded file)
         """
+        file_format = file_format.lower()
         url = self.get_datastore_dump_url(file_format)
         logger.debug(f"Downloading {url}")
         if filename is None:
             filename = self.data["name"]
-            extension = f".{file_format.lower()}"
+            extension = f".{file_format}"
             if not filename.endswith(extension):
                 filename = f"{filename}{extension}"
-        session = hdx.api.utilities.url_utils.get_ckan_ready_session(self.configuration)
-        with Download(session=session) as downloader:
-            if retriever:
-                downloader = retriever.clone(downloader)
-            path = downloader.download_file(url, folder=folder, filename=filename)
-            return url, path
+        return url, self._download_file(url, folder, filename, retriever)
 
     @staticmethod
     def get_all_resource_ids_in_datastore(
@@ -930,11 +1010,18 @@ class Resource(HDXObject):
             sort: Comma separated field names with optional asc/desc eg. "code desc". Defaults to None.
             distinct: Whether to return only distinct rows. Defaults to False.
             page_size: Number of rows to request per call. Defaults to 10000.
-            **kwargs: Other parameters to pass to datastore_search eg. plain, language.
+            **kwargs: Other parameters to pass to datastore_search eg. plain, language. records_format csv and tsv are not supported.
 
         Returns:
             Iterator of record dictionaries
         """
+        if page_size < 1:
+            raise HDXError("page_size must be at least 1!")
+        records_format = kwargs.get("records_format")
+        if records_format in ("csv", "tsv"):
+            raise HDXError(
+                f"records_format {records_format} is not supported! Use search_datastore instead."
+            )
         offset = 0
         while True:
             result = self.search_datastore(
@@ -950,9 +1037,11 @@ class Resource(HDXObject):
             )
             records = result["records"]
             yield from records
-            if len(records) < page_size:
+            # HDX caps limit at its maximum rows per call and returns the limit applied
+            limit = result["limit"]
+            if len(records) < limit:
                 return
-            offset += page_size
+            offset += limit
 
     def get_datastore_fields(self) -> list[dict]:
         """Get the field definitions of the resource datastore excluding the
@@ -971,7 +1060,8 @@ class Resource(HDXObject):
     ) -> list[dict]:
         """Run a read-only SQL query against the HDX datastore using CKAN
         datastore_search_sql. Tables are named by resource id, which must be
-        double quoted eg. SELECT * FROM "RESOURCE_ID".
+        double quoted eg. SELECT * FROM "RESOURCE_ID". Logs a warning if HDX
+        truncates the results.
 
         Args:
             sql: SQL SELECT statement
@@ -982,7 +1072,12 @@ class Resource(HDXObject):
         """
         resource = Resource(configuration=configuration)
         result = resource._write_to_hdx("datastore_search_sql", {"sql": sql})
-        return result["records"]
+        records = result["records"]
+        if result.get("records_truncated"):
+            logger.warning(
+                f"SQL query results truncated by HDX to {len(records)} rows!"
+            )
+        return records
 
     def delete_datastore(self) -> None:
         """Delete a resource from the HDX datastore
@@ -1000,6 +1095,41 @@ class Resource(HDXObject):
         if not success:
             logger.debug(result)
 
+    def submit_to_datapusher(self, ignore_hash: bool = False) -> bool:
+        """Submit the resource to DataPusher+ to load its file into the
+        datastore. This does not change last_modified. Unlike the automatic
+        submission on file upload, HDX's datastore allowlist and format checks
+        are not applied.
+
+        Args:
+            ignore_hash: Reload even if the file hash is unchanged. Defaults to False.
+
+        Returns:
+            True if submitted, False if not (eg. a job is already pending)
+        """
+        return self._write_to_hdx(
+            "datapusher_submit",
+            {"resource_id": self.data["id"], "ignore_hash": ignore_hash},
+            "resource_id",
+        )
+
+    def get_datapusher_status(self) -> dict | None:
+        """Get the status of the resource's latest DataPusher+ job
+
+        Returns:
+            Status dictionary with keys including status and error or None if no job
+        """
+        success, result = self._read_from_hdx(
+            "datapusher",
+            self.data["id"],
+            "resource_id",
+            self.actions()["datapusher_status"],
+        )
+        if success:
+            return result
+        logger.debug(result)
+        return None
+
     def get_resource_views(self) -> list[ResourceView]:
         """Get any resource views in the resource
 
@@ -1159,6 +1289,21 @@ class Resource(HDXObject):
         else:
             logger.debug(result)
 
+    def get_fs_check_info(self) -> list[dict]:
+        """Get file structure check information for the resource from its
+        fs_check_info field. Each check event can have keys such as state,
+        timestamp, sheet_changes and hxl_proxy_response.
+
+        Returns:
+            List of file structure check events
+        """
+        fs_check_info = self.data.get("fs_check_info")
+        if not fs_check_info:
+            return []
+        if isinstance(fs_check_info, str):
+            return json.loads(fs_check_info)
+        return fs_check_info
+
     def is_marked_data_updated(self) -> bool:
         """Return if the resource's data is marked to be updated
 

{hdx_python_api-6.7.2 → hdx_python_api-6.7.3}/src/hdx/data/user.py

@@ -45,6 +45,7 @@ class User(HDXObject):
             "list": "user_list",
             "listorgs": "organization_list_for_user",
             "token_list": "api_token_list",
+            "token_info": "hdx_token_info",
             "autocomplete": "user_autocomplete",
         }
 
@@ -334,7 +335,7 @@ class User(HDXObject):
         Returns:
             List of organization dicts in HDX that logged in user is a member of
         """
-        user = User(configuration=configuration)
+        user = cls(configuration=configuration)
         try:
             return user.configuration.call_remoteckan(
                 cls.actions()["listorgs"], {"permission": permission}
@@ -470,13 +471,25 @@ class User(HDXObject):
         Returns:
             List of Activity objects
         """
-        return Activity.get_activities(
-            "user",
-            self.data.get("id") or self.data.get("name"),
-            self.configuration,
-            all_pages,
-            **kwargs,
-        )
+        return self._get_activities("user", all_pages, **kwargs)
+
+    @classmethod
+    def get_current_token_info(cls, configuration: Configuration | None = None) -> dict:
+        """Get information about the API token in the configuration: the md5
+        hash of the token owner's email (email_hash) and the token name
+        (token_name).
+
+        Args:
+            configuration: HDX configuration. Defaults to global configuration.
+
+        Returns:
+            Dictionary with keys email_hash and token_name
+        """
+        user = cls(configuration=configuration)
+        try:
+            return user.configuration.call_remoteckan(cls.actions()["token_info"], {})
+        except Exception as e:
+            raise HDXError("Failed when trying to get API token info!") from e
 
     @classmethod
     def autocomplete(