hdx-python-api 6.6.7__tar.gz → 6.6.9__tar.gz

{hdx_python_api-6.6.7 → hdx_python_api-6.6.9}/.pre-commit-config.yaml

@@ -1,5 +1,5 @@
 default_language_version:
-  python: python3.13
+  python: python3
 
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks

hdx_python_api-6.6.9/CLAUDE.md

@@ -0,0 +1,57 @@
+# CLAUDE.md
+
+## Project Overview
+
+**hdx-python-api** is the official Python library for interacting with the [Humanitarian Data Exchange (HDX)](https://data.humdata.org/) platform. It provides ORM-like classes wrapping the CKAN API to read, create, update, and delete datasets, resources, organizations, users, vocabularies, showcases, and datastores.
+
+## Source Layout
+
+- `src/hdx/api/` — Configuration, session management, locations, utilities
+  - `configuration.py` — `Configuration` class (credentials, site URL, user agent)
+  - `remotehdx.py` — Low-level CKAN API client
+- `src/hdx/data/` — Data model objects, all inheriting from `HDXObject`
+  - `dataset.py`, `resource.py`, `organization.py`, `user.py`, `vocabulary.py`, `showcase.py`, `resource_view.py`
+  - `hdxobject.py` — Base class providing `_read_from_hdx` / `_write_to_hdx` and CRUD scaffolding
+- `src/hdx/facades/` — High-level entry-point helpers (`simple`, `keyword_arguments`, `infer_arguments`)
+- `tests/hdx/` — Test suite mirroring src layout; fixtures in `tests/fixtures/`
+
+## Running Tests
+
+```bash
+uv run pytest
+```
+
+Tests mock the CKAN HTTP session via inner `MockSession` classes in each test file — no live HDX connection needed for the standard suite. Integration tests require `HDX_KEY_TEST` and `HDX_PIPELINE_GSHEET_AUTH` environment variables.
+
+Coverage is written to `coverage.lcov` and JUnit XML to `test-results.xml`.
+
+## Code Style
+
+Formatted and linted with `ruff` (rules: E, F, I, UP; line-length not enforced). Run before committing:
+
+```bash
+pre-commit run --all-files
+```
+
+- Python ≥ 3.10
+- Type hints throughout; use `X | Y` union syntax (PEP 604), not `Optional`/`Union`
+- Google-style docstrings with `Args:` and `Returns:` sections
+- No inline comments unless the *why* is non-obvious
+
+## Key Patterns
+
+**HDXObject subclasses** expose a standard interface: `create_in_hdx()`, `update_in_hdx()`, `delete_from_hdx()`, `read_from_hdx()`. Each class defines an `actions()` static method mapping action names to CKAN API action strings.
+
+**Writing to HDX** uses `self._write_to_hdx(action_key, data_dict, id_field_name)` inherited from `HDXObject`. The `action_key` must be in `actions()`.
+
+**Test mocking** — each test fixture defines a `MockSession` with a `post()` method that inspects the URL and decoded JSON body to return `MockResponse` objects. State is tracked on class variables (e.g. `TestResource.datastore`).
+
+## Collaboration Style
+
+- Be objective, not agreeable. Act as a partner, not a sycophant. Push back when you disagree, flag tradeoffs honestly, and don't sugarcoat problems.
+- Keep explanations brief and to the point.
+- Don't rely on recalled knowledge for facts that could be stale (API behaviour, library versions, external systems). Search or read the actual source first. If you lack verified information, say so rather than speculate.
+
+## Scope of Changes
+
+When fixing a bug or addressing PR feedback, change only what is necessary to resolve the specific issue. Do not refactor surrounding code, rename variables, adjust formatting, or make improvements in the same commit unless they are directly required by the fix.

{hdx_python_api-6.6.7 → hdx_python_api-6.6.9}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: hdx-python-api
-Version: 6.6.7
+Version: 6.6.9
 Summary: HDX Python API for interacting with the Humanitarian Data Exchange
 Project-URL: Homepage, https://github.com/OCHA-DAP/hdx-python-api
 Author: Michael Rans
@@ -25,11 +25,11 @@ Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: 3.14
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Requires-Python: >=3.10
-Requires-Dist: ckanapi>=4.8
+Requires-Dist: ckanapi>=4.11
 Requires-Dist: defopt>=7.0.0
 Requires-Dist: email-validator
-Requires-Dist: hdx-python-country>=4.1.1
-Requires-Dist: hdx-python-utilities>=4.0.8
+Requires-Dist: hdx-python-country>=4.1.3
+Requires-Dist: hdx-python-utilities>=4.1.2
 Requires-Dist: makefun
 Requires-Dist: requests
 Provides-Extra: docs

{hdx_python_api-6.6.7 → hdx_python_api-6.6.9}/documentation/index.md

@@ -937,6 +937,33 @@ If the method **set_file_to_upload** is used to supply a file, the resource
 `last_modified` field is set to now automatically regardless of the value of
 `data_updated` or whether **mark_data_updated** has been called.
 
+### Data Dictionary
+
+A resource can have a data dictionary describing the columns in the data it
+contains. It is a list of dictionaries, each with the keys `field` (the
+column name as it appears in the file), `label` (a human-readable label) and
+`description` (a human-readable description of the column). Set it using
+**set_hdx_data_dictionary**:
+
+    resource.set_hdx_data_dictionary([
+        {
+            "field": "col_name",
+            "label": "Column Label",
+            "description": "Human-readable description.",
+        },
+        {
+            "field": "another_col",
+            "label": "Another Label",
+            "description": "Second column description.",
+        },
+    ])
+
+`field`, `label` and `description` must all be non-empty strings for every
+column, otherwise an `HDXError` is raised. Read the data dictionary back
+using the getter:
+
+    data_dictionary = resource.get_hdx_data_dictionary()
+
 ## Showcase Management
 
 The **Showcase** class enables you to manage showcases, creating, deleting and updating

{hdx_python_api-6.6.7 → hdx_python_api-6.6.9}/pyproject.toml

@@ -36,11 +36,11 @@ dynamic = ["version"]
 requires-python = ">=3.10"
 
 dependencies = [
-    "ckanapi>=4.8",
+    "ckanapi>=4.11",
     "defopt>=7.0.0",
     "email_validator",
-    "hdx-python-country>=4.1.1",
-    "hdx-python-utilities>=4.0.8",
+    "hdx-python-country>=4.1.3",
+    "hdx-python-utilities>=4.1.2",
     "makefun",
     "requests",
 ]

{hdx_python_api-6.6.7 → hdx_python_api-6.6.9}/src/hdx/api/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '6.6.7'
-__version_tuple__ = version_tuple = (6, 6, 7)
+__version__ = version = '6.6.9'
+__version_tuple__ = version_tuple = (6, 6, 9)
 
 __commit_id__ = commit_id = None

{hdx_python_api-6.6.7 → hdx_python_api-6.6.9}/src/hdx/data/resource.py

@@ -1,5 +1,6 @@
 """Resource class containing all logic for creating, checking, and updating resources."""
 
+import json
 import logging
 from collections.abc import Sequence
 from datetime import datetime
@@ -33,6 +34,9 @@ class Resource(HDXObject):
     """
 
     _formats_dict = None
+    _valid_data_dictionary_types = frozenset(
+        {"text", "numeric", "date", "timestamp without time zone"}
+    )
 
     def __init__(
         self,
@@ -62,6 +66,9 @@ class Resource(HDXObject):
             "search": "resource_search",
             "broken": "hdx_mark_broken_link_in_resource",
             "datastore_delete": "datastore_delete",
+            "datastore_create": "datastore_create",
+            "datastore_insert": "datastore_insert",
+            "datastore_upsert": "datastore_upsert",
             "datastore_search": "datastore_search",
         }
 
@@ -260,6 +267,56 @@ class Resource(HDXObject):
         self.data["format"] = file_format
         return file_format
 
+    def get_hdx_data_dictionary(self) -> list[dict] | None:
+        """Get the resource's data dictionary (column definitions)
+
+        Returns:
+            List of column definition dictionaries or None if not set
+        """
+        data_dictionary = self.data.get("hdx_data_dictionary")
+        if data_dictionary is None:
+            return None
+        if isinstance(data_dictionary, str):
+            return json.loads(data_dictionary)
+        return data_dictionary
+
+    def set_hdx_data_dictionary(self, data_dictionary: Sequence[dict]) -> None:
+        """Set the resource's data dictionary (column definitions). Each column
+        definition must be a dict with non-empty string values for field
+        (column name in the CSV file), label (human-readable column label) and
+        description (human-readable description of the column). If present,
+        data_type must be one of the PostgreSQL types produced by DataPusher+
+        (text, numeric, date, timestamp without time zone).
+
+        Args:
+            data_dictionary: Sequence of column definition dictionaries
+
+        Returns:
+            None
+        """
+        if not data_dictionary:
+            raise HDXError("hdx_data_dictionary must be a non-empty list!")
+        for i, column in enumerate(data_dictionary):
+            if not isinstance(column, dict):
+                raise HDXError(f"hdx_data_dictionary[{i}] must be a dict!")
+            for required_key in ("field", "label", "description"):
+                value = column.get(required_key)
+                if not isinstance(value, str) or not value.strip():
+                    raise HDXError(
+                        f"hdx_data_dictionary[{i}] is missing a non-empty '{required_key}'!"
+                    )
+            if "data_type" in column:
+                data_type = column["data_type"]
+                if data_type not in self._valid_data_dictionary_types:
+                    valid_types = ", ".join(sorted(self._valid_data_dictionary_types))
+                    raise HDXError(
+                        f"hdx_data_dictionary[{i}] has invalid 'data_type' "
+                        f"'{data_type}'! Must be one of: {valid_types}"
+                    )
+        self.data["hdx_data_dictionary"] = json.dumps(
+            data_dictionary, separators=(",", ":")
+        )
+
     def clean_format(self) -> str:
         """Clean the resource's format, setting it to None if it is invalid and
         cannot be mapped
@@ -701,6 +758,53 @@ class Resource(HDXObject):
                 return True
         return False
 
+    def create_datastore(
+        self,
+        schema: Sequence[dict],
+        primary_key: str | Sequence[str] | None = None,
+    ) -> None:
+        """Create a datastore for the resource with the given schema.
+
+        Args:
+            schema: Sequence of field definitions, each a dict with 'id' and 'type' keys.
+            primary_key: Primary key field name(s). Defaults to None.
+
+        Returns:
+            None
+        """
+        data: dict = {
+            "resource_id": self.data["id"],
+            "force": True,
+            "fields": schema,
+        }
+        if primary_key is not None:
+            if not isinstance(primary_key, str):
+                primary_key = ",".join(primary_key)
+            data["primary_key"] = primary_key
+        self._write_to_hdx("datastore_create", data, "resource_id")
+
+    def update_datastore(
+        self,
+        records: Sequence[dict],
+        method: str = "upsert",
+    ) -> None:
+        """Update (upsert) records into the resource datastore.
+
+        Args:
+            records: Sequence of record dicts to insert or update.
+            method: Datastore update method ('upsert', 'insert', or 'update'). Defaults to 'upsert'.
+
+        Returns:
+            None
+        """
+        data = {
+            "resource_id": self.data["id"],
+            "force": True,
+            "method": method,
+            "records": records,
+        }
+        self._write_to_hdx("datastore_upsert", data, "resource_id")
+
     def delete_datastore(self) -> None:
         """Delete a resource from the HDX datastore
 

{hdx_python_api-6.6.7 → hdx_python_api-6.6.9}/tests/hdx/api/utilities/test_hdx_state.py

@@ -47,7 +47,7 @@ class TestState:
     def do_state(self, tempfolder, statefile):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 if "resource" in url:
                     result = json.dumps(resultdict)
                     return MockResponse(
@@ -71,7 +71,7 @@ class TestState:
     def do_state_multi(self, tempfolder, multidatestatefile):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 if "resource" in url:
                     result = json.dumps(resultdict)
                     return MockResponse(

{hdx_python_api-6.6.7 → hdx_python_api-6.6.9}/tests/hdx/data/test_dataset_add_hapi_error.py

@@ -13,7 +13,7 @@ class TestDatasetAddHAPIError:
     def hapi_resource_update(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 if "show" in url:
                     resource_id = datadict["id"]

{hdx_python_api-6.6.7 → hdx_python_api-6.6.9}/tests/hdx/data/test_dataset_core.py

@@ -182,7 +182,7 @@ class TestDatasetCore:
     def read(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 return dataset_mockshow(url, datadict)
 
@@ -192,7 +192,7 @@ class TestDatasetCore:
     def post_revise(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 if isinstance(data, dict):
                     datadict = {
                         k.decode("utf8"): v.decode("utf8") for k, v in data.items()
@@ -219,7 +219,7 @@ class TestDatasetCore:
     def post_create(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 if isinstance(data, dict):
                     datadict = {
                         k.decode("utf8"): v.decode("utf8") for k, v in data.items()
@@ -286,7 +286,7 @@ class TestDatasetCore:
     def post_update(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 if isinstance(data, dict):
                     datadict = {
                         k.decode("utf8"): v.decode("utf8") for k, v in data.items()
@@ -363,7 +363,7 @@ class TestDatasetCore:
     def post_reorder(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 decodedata = data.decode("utf-8")
                 datadict = json.loads(decodedata)
                 if "show" in url:
@@ -390,7 +390,7 @@ class TestDatasetCore:
     def post_delete(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 decodedata = data.decode("utf-8")
                 datadict = json.loads(decodedata)
                 if "show" in url:
@@ -423,7 +423,7 @@ class TestDatasetCore:
     def search(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 return mocksearch(url, datadict)
 
@@ -433,7 +433,7 @@ class TestDatasetCore:
     def post_list(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 return mocklist(url, datadict)
 
@@ -443,7 +443,7 @@ class TestDatasetCore:
     def all(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 return mockall(url, datadict)
 
@@ -453,7 +453,7 @@ class TestDatasetCore:
     def post_autocomplete(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 decodedata = data.decode("utf-8")
                 datadict = json.loads(decodedata)
                 if "autocomplete" not in url or "acled" not in datadict["q"]:

{hdx_python_api-6.6.7 → hdx_python_api-6.6.9}/tests/hdx/data/test_dataset_noncore.py

@@ -52,7 +52,7 @@ class TestDatasetNoncore:
 
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 return vocabulary_mockshow(url, datadict)
 
@@ -62,7 +62,7 @@ class TestDatasetNoncore:
     def user_read(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 return user_mockshow(url, datadict)
 
@@ -72,7 +72,7 @@ class TestDatasetNoncore:
     def organization_read(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 return organization_mockshow(url, datadict)
 
@@ -82,7 +82,7 @@ class TestDatasetNoncore:
     def showcase_read(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 if "showcase_list" in url:
                     result = json.dumps([showcase_resultdict])
@@ -111,7 +111,7 @@ class TestDatasetNoncore:
     def vocabulary_update(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 if isinstance(data, dict):
                     datadict = {
                         k.decode("utf8"): v.decode("utf8") for k, v in data.items()

{hdx_python_api-6.6.7 → hdx_python_api-6.6.9}/tests/hdx/data/test_organization.py

@@ -115,7 +115,7 @@ class TestOrganization:
     def read(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 return organization_mockshow(url, datadict)
 
@@ -125,7 +125,7 @@ class TestOrganization:
     def post_create(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 if "show" in url:
                     return organization_mockshow(url, datadict)
@@ -165,7 +165,7 @@ class TestOrganization:
     def post_update(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 if "show" in url:
                     return organization_mockshow(url, datadict)
@@ -205,7 +205,7 @@ class TestOrganization:
     def post_delete(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 decodedata = data.decode("utf-8")
                 datadict = json.loads(decodedata)
                 if "show" in url:
@@ -232,7 +232,7 @@ class TestOrganization:
     def post_list(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 json.loads(data.decode("utf-8"))
                 return mocklist(url)
 
@@ -242,7 +242,7 @@ class TestOrganization:
     def post_all_fields(self, fixturesfolder):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 kwargs = json.loads(data.decode("utf-8"))
                 if "show" in url:
                     return organization_mockshow(url, kwargs)
@@ -264,7 +264,7 @@ class TestOrganization:
     def user_read(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 return user_mockshow(url, datadict)
 
@@ -274,7 +274,7 @@ class TestOrganization:
     def datasets_get(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 return mockgetdatasets(url, datadict)
 
@@ -284,7 +284,7 @@ class TestOrganization:
     def post_autocomplete(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 decodedata = data.decode("utf-8")
                 datadict = json.loads(decodedata)
                 if "autocomplete" not in url or "innago" not in datadict["q"]: