hdx-python-api 6.7.2__tar.gz → 6.7.4__tar.gz

{hdx_python_api-6.7.2 → hdx_python_api-6.7.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: hdx-python-api
-Version: 6.7.2
+Version: 6.7.4
 Summary: HDX Python API for interacting with the Humanitarian Data Exchange
 Project-URL: Homepage, https://github.com/OCHA-DAP/hdx-python-api
 Author: Michael Rans

{hdx_python_api-6.7.2 → hdx_python_api-6.7.4}/documentation/index.md

@@ -36,6 +36,7 @@ upload your datasets to HDX.
     -   [Location Management](#location-management)
     -   [Vocabulary Management](#vocabulary-management)
     -   [Activity Streams](#activity-streams)
+    -   [HDX Statistics](#hdx-statistics)
     -   [Pipeline State](#pipeline-state)
 -   [Working Examples](#working-examples)
 -   [Project Framework](#project-framework)
@@ -636,6 +637,45 @@ function, for example:
 
     dataset.remove_showcase(showcase)
 
+HDX runs file structure checks on resources, recording schema changes (eg.
+column names, order or types) and whether sheets have merged cells. You can get
+these checks for all of a dataset's resources from HDX:
+
+    fs_check_info = dataset.get_fs_check_info()
+
+This returns a list of dictionaries, each from a resource name to a list of
+check events with keys such as `state`, `timestamp`, `sheet_changes` and
+`hxl_proxy_response`.
+
+Similarly, you can get the geopreview shape information of a dataset's
+resources:
+
+    shape_info = dataset.get_shape_info()
+
+HDX can recommend tags for a dataset based on its title and organization
+(`owner_org`). Tags are ordered from most to least recommended:
+
+    tags = dataset.get_recommended_tags()
+
+You can check if the dataset or its organization is in HDX's allowlist for
+automatic loading of resources into the datastore:
+
+    allowed = dataset.is_allowed_for_datastore()
+
+HDX sysadmins can submit all of a dataset's CSV resources to DataPusher+
+(returning False if there are none):
+
+    submitted = dataset.push_to_datastore()
+
+Users who are sysadmins or have permission to manage QA can mark QA of a
+dataset as completed, set a QA field of all its resources or clear a file
+structure check field (`fs_check_info` by default) of all its resources. The
+dataset is updated from the result:
+
+    dataset.set_qa_completed(True)
+    dataset.qa_revise_resources("pii", "false")
+    dataset.reset_fs_check()
+
 ### Time Period
 
 Time Period is a mandatory field in HDX. It is the earliest start date and
@@ -947,6 +987,38 @@ If the method **set_file_to_upload** is used to supply a file, the resource
 `last_modified` field is set to now automatically regardless of the value of
 `data_updated` or whether **mark_data_updated** has been called.
 
+The file structure check events of a resource (see Dataset Specific
+Operations) can be obtained from its `fs_check_info` field without a call to
+HDX:
+
+    events = resource.get_fs_check_info()
+
+Users who are sysadmins or have permission to manage QA can set or clear a
+file structure check field of a resource. If the key is `fs_check_info` and the
+value is a dictionary, it is appended to the existing check events (the last 20
+are kept):
+
+    resource.revise_fs_check("fs_check_info", {"state": "success", ...})
+    resource.reset_fs_check()  # key defaults to fs_check_info
+
+They can also quarantine a resource (which removes its `shape_info` and
+geopreview) or take it out of quarantine, and patch other QA fields:
+
+    in_quarantine = resource.is_in_quarantine()
+    resource.set_in_quarantine(True)
+    resource.qa_patch(pii="true", sdc_report_flag="false")
+
+A resource can be flagged as containing p-codes or as being in HAPI. Setting
+these flags requires sysadmin or permission to manage p-codes or the in HAPI
+flag respectively:
+
+    p_coded = resource.is_p_coded()  # None if not set
+    resource.set_p_coded(True)
+    in_hapi = resource.is_in_hapi()
+    resource.set_in_hapi(True)
+
+These methods update the resource from the result.
+
 ### Data Dictionary
 
 A resource can have a data dictionary describing the columns in the data it
@@ -974,11 +1046,36 @@ using the getter:
 
     data_dictionary = resource.get_hdx_data_dictionary()
 
+To change the data dictionary of a resource already in HDX without sending
+dataset update notifications, use **update_hdx_data_dictionary**. It patches
+only the `hdx_data_dictionary` field (so `last_modified` is unchanged), then,
+if the resource has a datastore, updates the datastore field labels and
+descriptions without reloading the data. The given columns are merged by
+`field` into the existing data dictionary in the resource metadata, so read
+the resource from HDX first. Pass `replace=True` to replace the data
+dictionary instead and `update_datastore=False` to leave the datastore alone:
+
+    resource = Resource.read_from_hdx(RESOURCE_ID)
+    resource.update_hdx_data_dictionary([
+        {
+            "field": "col_name",
+            "label": "Column Label",
+            "description": "New description.",
+        },
+    ])
+
+Columns are matched to datastore fields by `field`. A warning is logged for
+any column with no matching datastore field.
+
 ### Datastore
 
-A resource's data can be loaded into the HDX datastore and queried. HDX
-requires an API key for all datastore actions and dumps, including reads of
-public resources. To create, update or delete the datastore for a resource:
+A resource's data can be loaded into the HDX datastore and queried. HDX calls this
+[Tabular Data Endpoints (TDE)](https://docs.humdata.org/build/hdx-apis/tabular-data-endpoints),
+and it is available only for resources with TDE enabled (their `datastore_active`
+field is true). Contact hdx@un.org to request TDE for your datasets. HDX requires an
+API key for all datastore actions and dumps, including reads of public resources,
+and limits each IP address to 60 requests per minute. To create, update or delete
+the datastore for a resource:
 
     resource.create_datastore(
         [{"id": "code", "type": "text"}, {"id": "value", "type": "numeric"}],
@@ -988,6 +1085,22 @@ public resources. To create, update or delete the datastore for a resource:
     has_datastore = resource.has_datastore()
     resource.delete_datastore()
 
+HDX loads uploaded files into the datastore using DataPusher+. To reload a
+resource's file into the datastore without uploading it again (so
+`last_modified` is unchanged), use **submit_to_datapusher**. It returns False
+if the job was not submitted, eg. because one is already pending. HDX's
+datastore allowlist and format checks, which apply to automatic submissions
+on upload, are not applied. **get_datapusher_status** returns the status of
+the latest job or None if there is none:
+
+    submitted = resource.submit_to_datapusher(ignore_hash=True)
+    status = resource.get_datapusher_status()  # eg. status["status"] == "complete"
+
+HDX sysadmins can instead use **push_to_datastore**, which submits the
+resource through HDX's wrapper action:
+
+    submitted = resource.push_to_datastore()
+
 To download the whole datastore (rather than the resource's file) use
 **download_datastore**, which returns the URL and path like **download**.
 `file_format` can be `csv` (default), `tsv`, `json` or `xml`, and `filename`
@@ -1008,7 +1121,8 @@ and `include_total`, and passes any other keyword arguments to CKAN:
     records = result["records"]
 
 **get_datastore_records** takes the same query parameters but pages through
-all matching rows, yielding each record (`page_size` defaults to 10000):
+all matching rows, yielding each record (`page_size` defaults to 10000).
+`records_format` csv and tsv are not supported; use `search_datastore` for those:
 
     for record in resource.get_datastore_records(filters={"code": "A"}):
         ...
@@ -1020,7 +1134,9 @@ internal `_id` field:
 
 **search_datastore_sql** wraps CKAN's `datastore_search_sql`, running a
 SELECT statement where tables are named by double quoted resource id and
-returning the list of records:
+returning the list of records. HDX returns at most 32,000 rows per query and
+cuts off larger results, in which case a warning is logged (see
+[paging in the TDE docs](https://docs.humdata.org/build/hdx-apis/tabular-data-endpoints/troubleshooting-and-error-handling#pagination)):
 
     records = Resource.search_datastore_sql(
         'SELECT code, SUM(value) AS total FROM "RESOURCE_ID" GROUP BY code'
@@ -1067,6 +1183,11 @@ configuration):
 
     user = User.get_current_user()
 
+You can get the name of the API token in the configuration and the md5 hash
+of its owner's email:
+
+    token_info = User.get_current_token_info()  # {"email_hash": ..., "token_name": ...}
+
 You can check that the current user has a particular permission to a specific
 organization:
 
@@ -1152,6 +1273,17 @@ since Data Grid membership changes over time, rather than a fixed list):
 
 This returns a sorted list of 3-letter location codes.
 
+HDX sysadmins can get the Data Grid of a country and clear HDX's cached copy of
+it:
+
+    data_grid = Location.get_data_grid("AFG")
+    Location.invalidate_data_grid("AFG")
+
+**get_data_grid** returns a dictionary with keys including `title`, `date`,
+`stats` and `categories` (or None if not found). Each category has
+`subcategories`, each with a list of `datasets`. Complementary datasets have
+`is_complementary` set to True.
+
 You can get the datasets belonging to a location as follows:
 
     datasets = location.get_datasets(**kwargs)
@@ -1223,6 +1355,28 @@ permission to edit the object the activity relates to for these:
     data = activity.get_data()
     diff = activity.get_diff(diff_type="unified")  # or "context" or "html"
 
+## HDX Statistics
+
+General statistics about datasets, organizations and users on HDX can be
+obtained as follows:
+
+    from hdx.api.utilities.hdx_stats import get_general_statistics, push_general_stats
+
+    stats = get_general_statistics()
+
+Users with permission to manage basic scheduled tasks can compute the statistics
+and push them to analytics:
+
+    pushed_stats = push_general_stats()
+
+HDX sysadmins can get the number of activities of each active user (optionally
+between start and end dates) and lists of active and inactive organizations:
+
+    from hdx.api.utilities.hdx_stats import get_organization_statistics, get_user_statistics
+
+    user_stats = get_user_statistics(start_date="2026-01-01", end_date="2026-06-30")
+    org_stats = get_organization_statistics()
+
 ## Pipeline State
 
 The HDXState class allows the reading and writing of state to a given dataset. Input

{hdx_python_api-6.7.2 → hdx_python_api-6.7.4}/src/hdx/api/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '6.7.2'
-__version_tuple__ = version_tuple = (6, 7, 2)
+__version__ = version = '6.7.4'
+__version_tuple__ = version_tuple = (6, 7, 4)
 
 __commit_id__ = commit_id = None

hdx_python_api-6.7.4/src/hdx/api/utilities/hdx_stats.py

@@ -0,0 +1,86 @@
+"""HDX site-wide statistics"""
+
+from hdx.api.configuration import Configuration
+from hdx.data.hdxobject import HDXError
+
+
+def _call_stats_action(
+    action: str, configuration: Configuration | None, data: dict | None = None
+) -> dict:
+    """Call an HDX statistics action
+
+    Args:
+        action: CKAN action name
+        configuration: HDX configuration. Defaults to global configuration.
+        data: Parameters to pass to the action. Defaults to None.
+
+    Returns:
+        Statistics dictionary
+    """
+    if configuration is None:
+        configuration = Configuration.read()
+    try:
+        return configuration.call_remoteckan(action, data or {})
+    except Exception as e:
+        raise HDXError(f"Failed when trying to call {action}!") from e
+
+
+def get_general_statistics(configuration: Configuration | None = None) -> dict:
+    """Get general statistics about datasets, organizations and users on HDX
+
+    Args:
+        configuration: HDX configuration. Defaults to global configuration.
+
+    Returns:
+        Dictionary with keys datasets, organizations and users
+    """
+    return _call_stats_action("hdx_general_statistics", configuration)
+
+
+def push_general_stats(configuration: Configuration | None = None) -> dict:
+    """Compute general statistics on HDX and push them to analytics. Requires
+    permission to manage basic scheduled tasks.
+
+    Args:
+        configuration: HDX configuration. Defaults to global configuration.
+
+    Returns:
+        Statistics that were pushed
+    """
+    return _call_stats_action("hdx_push_general_stats", configuration)
+
+
+def get_user_statistics(
+    start_date: str | None = None,
+    end_date: str | None = None,
+    configuration: Configuration | None = None,
+) -> dict:
+    """Get the number of activities of each active user on HDX, optionally
+    between start and end dates. Requires sysadmin.
+
+    Args:
+        start_date: Count activities on or after this date eg. 2026-01-01. Defaults to None.
+        end_date: Count activities on or before this date. Defaults to None.
+        configuration: HDX configuration. Defaults to global configuration.
+
+    Returns:
+        Dictionary from user id to dictionary with keys id, name and no_of_activities
+    """
+    data = {}
+    if start_date:
+        data["start_date"] = start_date
+    if end_date:
+        data["end_date"] = end_date
+    return _call_stats_action("hdx_user_statistics", configuration, data)
+
+
+def get_organization_statistics(configuration: Configuration | None = None) -> dict:
+    """Get active and inactive organizations on HDX. Requires sysadmin.
+
+    Args:
+        configuration: HDX configuration. Defaults to global configuration.
+
+    Returns:
+        Dictionary with keys active_organizations and inactive_organizations
+    """
+    return _call_stats_action("hdx_organization_statistics", configuration)

{hdx_python_api-6.7.2 → hdx_python_api-6.7.4}/src/hdx/data/activity.py

@@ -129,7 +129,9 @@ class Activity(HDXObject):
             result = activity._write_to_hdx(action, data)
             return [cls(x, configuration=configuration) for x in result]
         data = dict(data)
-        limit = data.setdefault("limit", 100)
+        # HDX silently caps limit at ckan.activity_list_limit_max
+        limit = min(data.get("limit", 100), 100)
+        data["limit"] = limit
         offset = data.get("offset", 0)
         activities = []
         while True:

{hdx_python_api-6.7.2 → hdx_python_api-6.7.4}/src/hdx/data/dataset.py

@@ -147,6 +147,14 @@ class Dataset(HDXObject):
             "list": "package_list",
             "autocomplete": "package_autocomplete",
             "create_default_views": "package_create_default_resource_views",
+            "fs_check_info": "fs_check_info_show",
+            "fs_check_reset": "hdx_fs_check_package_reset",
+            "shape_info": "shape_info_show",
+            "recommend_tags": "hdx_recommend_tags",
+            "allowed_for_datastore": "hdx_is_package_allowed_for_datastore",
+            "push_to_datastore": "hdx_push_resource_to_datastore",
+            "qa_completed": "hdx_mark_qa_completed",
+            "qa_revise_resources": "hdx_qa_package_revise_resource",
         }
 
     def __setitem__(self, key: Any, value: Any) -> None:
@@ -1447,13 +1455,7 @@ class Dataset(HDXObject):
         Returns:
             List of Activity objects
         """
-        return Activity.get_activities(
-            "package",
-            self.data.get("id") or self.data.get("name"),
-            self.configuration,
-            all_pages,
-            **kwargs,
-        )
+        return self._get_activities("package", all_pages, **kwargs)
 
     @classmethod
     def autocomplete(
@@ -2333,6 +2335,155 @@ class Dataset(HDXObject):
             return None
         return f"{self.configuration.get_hdx_site_url()}/api/3/action/package_show?id={name_or_id}"
 
+    def get_fs_check_info(self) -> list[dict]:
+        """Get file structure check information for the dataset's resources
+        from HDX. Each list item is a dictionary from resource name to a list
+        of check events, each of which can have keys such as state, timestamp,
+        sheet_changes and hxl_proxy_response.
+
+        Returns:
+            List of dictionaries from resource name to file structure checks
+        """
+        success, result = self._read_from_hdx(
+            "dataset",
+            self.get_name_or_id(prefer_name=False),
+            "id",
+            self.actions()["fs_check_info"],
+        )
+        if success:
+            return result
+        return []
+
+    def _set_from_package(self, package: dict) -> None:
+        """Set the dataset's metadata and resources from a package metadata
+        dictionary returned by HDX
+
+        Args:
+            package: Package metadata dictionary
+
+        Returns:
+            None
+        """
+        self.data = package
+        self.init_resources()
+        self.separate_resources()
+
+    def reset_fs_check(self, key: str = "fs_check_info") -> None:
+        """Clear a file structure check field of all the dataset's resources
+        in HDX. Requires sysadmin or permission to manage QA.
+
+        Args:
+            key: Resource field to clear. Defaults to fs_check_info.
+
+        Returns:
+            None
+        """
+        result = self._write_to_hdx(
+            "fs_check_reset", {"package_id": self.data["id"], "key": key}, "package_id"
+        )
+        self._set_from_package(result["package"])
+
+    def set_qa_completed(self, qa_completed: bool) -> None:
+        """Mark QA of the dataset in HDX as completed or not. Requires
+        sysadmin or permission to manage QA.
+
+        Args:
+            qa_completed: Whether QA is completed
+
+        Returns:
+            None
+        """
+        result = self._write_to_hdx(
+            "qa_completed", {"id": self.data["id"], "qa_completed": qa_completed}, "id"
+        )
+        self._set_from_package(result["package"])
+
+    def qa_revise_resources(self, key: str, value: Any) -> None:
+        """Set a QA field of all the dataset's resources in HDX. Requires
+        sysadmin or permission to manage QA.
+
+        Args:
+            key: Resource field eg. pii
+            value: Value to set
+
+        Returns:
+            None
+        """
+        result = self._write_to_hdx(
+            "qa_revise_resources",
+            {"id": self.data["id"], "key": key, "value": value},
+            "id",
+        )
+        self._set_from_package(result["package"])
+
+    def get_shape_info(self) -> list[dict]:
+        """Get geopreview shape information for the dataset's resources from
+        HDX
+
+        Returns:
+            List of dictionaries from resource name to shape information
+        """
+        success, result = self._read_from_hdx(
+            "dataset",
+            self.get_name_or_id(prefer_name=False),
+            "id",
+            self.actions()["shape_info"],
+        )
+        if success:
+            return result
+        return []
+
+    def get_recommended_tags(self) -> list[str]:
+        """Get tags recommended by HDX for the dataset based on its title and
+        organization. Tags are ordered from most to least recommended.
+
+        Returns:
+            List of recommended tags
+        """
+        data = {"title": self.data.get("title")}
+        organization = self.data.get("owner_org")
+        if organization:
+            data["organization"] = organization
+        try:
+            result = self.configuration.call_remoteckan(
+                self.actions()["recommend_tags"], data
+            )
+        except Exception as e:
+            raise HDXError("Failed when trying to get recommended tags!") from e
+        return [tag["name"] for tag in result]
+
+    def is_allowed_for_datastore(self) -> bool:
+        """Return if the dataset or its organization is in HDX's allowlist for
+        automatic loading of resources into the datastore
+
+        Returns:
+            Whether dataset is allowed for datastore
+        """
+        success, result = self._read_from_hdx(
+            "dataset",
+            self.data["id"],
+            "package_id",
+            self.actions()["allowed_for_datastore"],
+        )
+        if success:
+            return result
+        logger.debug(result)
+        return False
+
+    def push_to_datastore(self) -> bool:
+        """Submit all the dataset's CSV resources to DataPusher+ using HDX's
+        wrapper action. Requires sysadmin.
+
+        Returns:
+            True if submitted, False if not (eg. no CSV resources)
+        """
+        result = self._write_to_hdx(
+            "push_to_datastore", {"dataset_id": self.data["id"]}, "dataset_id"
+        )
+        if not result["success"]:
+            logger.debug(result["message"])
+        return result["success"]
+
     def generate_resource(
         self,
         folder: Path | str,

{hdx_python_api-6.7.2 → hdx_python_api-6.7.4}/src/hdx/data/hdxobject.py

@@ -9,7 +9,7 @@ from collections import UserDict
 from collections.abc import Sequence
 from os.path import isfile
 from pathlib import Path
-from typing import Any, Optional, Union
+from typing import TYPE_CHECKING, Any, Optional, Union
 
 from ckanapi.errors import NotFound
 from hdx.utilities.dictandlist import merge_two_dictionaries
@@ -20,6 +20,9 @@ from hdx.utilities.loader import (
 
 from hdx.api.configuration import Configuration
 
+if TYPE_CHECKING:
+    from hdx.data.activity import Activity
+
 logger = logging.getLogger(__name__)
 
 
@@ -376,6 +379,30 @@ class HDXObject(UserDict, ABC):
             **kwargs,
         )
 
+    def _get_activities(
+        self, object_type: str, all_pages: bool = False, **kwargs: Any
+    ) -> list["Activity"]:
+        """Helper method to get the activity stream of this object
+
+        Args:
+            object_type: One of package, organization, group or user
+            all_pages: Whether to get all pages of results. Defaults to False.
+            **kwargs: Parameters to pass to Activity.get_activities
+
+        Returns:
+            List of Activity objects
+        """
+        # Imported here as hdx.data.activity imports this module
+        from hdx.data.activity import Activity
+
+        return Activity.get_activities(
+            object_type,
+            self.data.get("id") or self.data.get("name"),
+            self.configuration,
+            all_pages,
+            **kwargs,
+        )
+
     def _write_to_hdx(
         self,
         action: str,

{hdx_python_api-6.7.2 → hdx_python_api-6.7.4}/src/hdx/data/location.py

@@ -43,6 +43,8 @@ class Location(HDXObject):
             "delete": "group_delete",
             "list": "group_list",
             "autocomplete": "group_autocomplete",
+            "data_grid": "hdx_datagrid_show",
+            "invalidate_data_grid": "invalidate_data_completeness_for_location",
         }
 
     @classmethod
@@ -218,6 +220,47 @@ class Location(HDXObject):
             if len(group["name"]) == 3 and group.get("data_completeness") == "active"
         )
 
+    @staticmethod
+    def get_data_grid(
+        iso3: str, configuration: Configuration | None = None
+    ) -> dict | None:
+        """Get the Data Grid of a country. The dictionary has keys including
+        title, date, stats and categories. Each category has subcategories,
+        each with a list of datasets. Complementary datasets have
+        is_complementary set to True. Requires sysadmin.
+
+        Args:
+            iso3: Country ISO 3-letter code
+            configuration: HDX configuration. Defaults to global configuration.
+
+        Returns:
+            Data Grid dictionary or None if not found
+        """
+        location = Location(configuration=configuration)
+        success, result = location._read_from_hdx(
+            "location", iso3.lower(), "id", location.actions()["data_grid"]
+        )
+        if success:
+            return result
+        logger.debug(result)
+        return None
+
+    @staticmethod
+    def invalidate_data_grid(
+        iso3: str, configuration: Configuration | None = None
+    ) -> None:
+        """Clear HDX's cached Data Grid of a country. Requires sysadmin.
+
+        Args:
+            iso3: Country ISO 3-letter code
+            configuration: HDX configuration. Defaults to global configuration.
+
+        Returns:
+            None
+        """
+        location = Location(configuration=configuration)
+        location._write_to_hdx("invalidate_data_grid", {"name": iso3.lower()}, "name")
+
     def get_activities(self, all_pages: bool = False, **kwargs: Any) -> list[Activity]:
         """Get the activity stream of this location. One page of results (whose size
         is given by limit) is returned unless all_pages is True.
@@ -232,13 +275,7 @@ class Location(HDXObject):
         Returns:
             List of Activity objects
         """
-        return Activity.get_activities(
-            "group",
-            self.data.get("id") or self.data.get("name"),
-            self.configuration,
-            all_pages,
-            **kwargs,
-        )
+        return self._get_activities("group", all_pages, **kwargs)
 
     @classmethod
     def autocomplete(

{hdx_python_api-6.7.2 → hdx_python_api-6.7.4}/src/hdx/data/organization.py

@@ -326,13 +326,7 @@ class Organization(HDXObject):
         Returns:
             List of Activity objects
         """
-        return Activity.get_activities(
-            "organization",
-            self.data.get("id") or self.data.get("name"),
-            self.configuration,
-            all_pages,
-            **kwargs,
-        )
+        return self._get_activities("organization", all_pages, **kwargs)
 
     @classmethod
     def autocomplete(