pymetadataeditor 0.3.2__tar.gz → 0.5.0__tar.gz

pymetadataeditor-0.5.0/PKG-INFO

@@ -0,0 +1,215 @@
+Metadata-Version: 2.4
+Name: pymetadataeditor
+Version: 0.5.0
+Summary: Metadata Editor client for python
+License: MIT
+License-File: LICENSE
+Author: Gordon Blackadder
+Author-email: gblackadder@worldbank.org
+Requires-Python: >=3.11,<4.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3.15
+Provides-Extra: llm
+Requires-Dist: markitdown (>=0.0.1a3,<0.0.2) ; extra == "llm"
+Requires-Dist: metadataschemas (>=0.1.3,<0.2.0)
+Requires-Dist: openai (>=2.8.0,<3.0.0) ; extra == "llm"
+Requires-Dist: pandas (>=2.2.2,<3.0.0)
+Requires-Dist: pydantic (>=2.7.4,<3.0.0)
+Requires-Dist: requests (>=2.32.3,<3.0.0)
+Requires-Dist: tiktoken (>=0.8.0) ; extra == "llm"
+Requires-Dist: tqdm ; extra == "llm"
+Requires-Dist: urllib3 (>=2.2.2,<3.0.0)
+Description-Content-Type: text/markdown
+
+# pyMetadataEditor
+
+A Python client for [Metadata Editor](https://github.com/worldbank/metadata-editor): create, edit, and manage metadata for microdata, indicators, geospatial data, documents, scripts, images, and videos.
+
+## Installation
+
+This project uses [Poetry](https://python-poetry.org/docs/basic-usage/):
+
+```bash
+pip install poetry
+cd pymetadataeditor
+poetry install
+```
+
+The LLM-assisted metadata features (OpenAI, `tiktoken`, `markitdown`) are optional: `poetry install --extras llm`, or `pip install "pymetadataeditor[llm]"`.
+
+For development (pytest, pre-commit):
+
+```bash
+poetry install --with dev
+poetry run pre-commit install
+```
+
+## Quick start
+
+```python
+from pymetadataeditor import MetadataEditor
+import os
+
+me = MetadataEditor(
+    api_url=os.getenv("METADATA_API_URL"),   # e.g. https://host/index.php/api
+    api_key=os.getenv("METADATA_API_KEY"),
+    verify_ssl=False,  # set True when SSL is verified on your server
+)
+```
+
+### List projects
+
+`list` returns one page (summary fields, no metadata) plus the server `total`:
+
+```python
+response = me.projects.list(limit=10, metadata_type="indicator", keywords="literacy rate")
+print(response["total"])
+for project in response["projects"]:
+    print(project["id"], project["type"], project["title"])
+```
+
+Filters: `keywords`, `metadata_type`, `collection` (+ `exclude_collections=True`), `tag`, `ownership`, `users`,
+`created_by`, `changed_by`, `date_start` / `date_end` (last changed), `sort_by`. List filters accept Python lists.
+`me.projects.facets()` returns the available types, collections and their counts.
+
+To go through every matching project, iterate instead of loading everything at once:
+
+```python
+for project in me.projects.iter_all(metadata_type="indicator", page_size=500):
+    ...
+
+me.projects.count(collection=12)                # just the number
+me.projects.list_all(metadata_type="table")     # everything in memory: {"projects": [...], "total": n}
+```
+
+`iter_all` pages in id order, so projects edited while you iterate are neither skipped nor repeated. Other large
+listings have the same pattern: `me.collections.iter_projects`, `me.jobs.iter_all`, `me.codelists.iter_all`,
+`me.codelists.iter_codes`, `me.data_structures.iter_all`, `me.indicators.iter_data_rows`.
+
+### Create a project (template-aware)
+
+```python
+demo_name = "my_indicator_demo"
+metadata = {
+    "idno": demo_name,
+    "series_description": {
+        "idno": demo_name,
+        "name": "Version 1",
+    },
+}
+
+project_id = me.metadata.create(
+    metadata,
+    metadata_type_or_template_uid="indicator",
+)
+```
+
+### Read metadata back
+
+```python
+model = me.metadata.get(project_id, output_mode="pydantic")
+model.pretty_print()
+```
+
+## API overview (v2)
+
+Thin clients return **raw API JSON** (`dict`). Workflow helpers on `me.metadata` handle validation and Pydantic/Excel conversion.
+
+| Client | Role |
+|--------|------|
+| `me.projects` | List, get, create, update, patch, delete projects |
+| `me.metadata` | Template-aware create, update, get (dict / pydantic / excel) |
+| `me.collections` | Collections and membership |
+| `me.templates` | Template CRUD |
+| `me.resources` | External resources (reports, questionnaires, files) |
+| `me.jobs` | Background jobs (publish to NADA, indicator data import, poll, list) |
+| `me.publish` | Publish metadata and indicator DSD/data to NADA (synchronous) |
+| `me.uploads` | Resumable (chunked) file uploads |
+| `me.data_structures` | Data structure (DSD) registry: structures, components, CSV/JSON/SDMX import, export |
+| `me.codelists` | Codelist registry: codelists, codes, translations, import/export |
+| `me.indicators` | Indicator projects: bind a DSD, load/export data, chart data, `notes`, publish |
+| `me.api` | Generic `get` / `post` / `put` / `delete` for any path |
+
+API errors raise `ApiError` (`status_code`, `message`, `errors`), a subclass of `requests` `HTTPError`.
+
+Methods still on `me` directly include `make_metadata_outline`, `read_metadata_from_excel`, `draft_metadata_from_files`, and related helpers.
+
+**Breaking change:** v1 flat methods (`list_projects`, `create_project_log`, …) were removed. See [legacy-migration.MD](legacy-migration.MD).
+
+### Direct HTTP (`me.api`)
+
+For endpoints without a typed wrapper yet:
+
+```python
+me.api.get("/editor", params={"limit": 5})
+me.api.post("/collections/add_projects", json={"collections": [1], "id_format": "id", "projects": [101]})
+```
+
+POST body styles:
+
+- `json={...}` — JSON API payloads
+- `data={...}` — form fields
+- `data={...}` + `files={...}` — multipart uploads
+
+Many deletes use **POST** in restricted networks (e.g. `me.projects.delete`, `me.collections.delete`), not HTTP DELETE.
+
+### Publish to NADA
+
+`me.publish.*` publishes now and returns NADA's answers; `me.jobs.queue_*` methods queue a job on the
+editor's worker and return its `uuid` (pass `wait=True` to block until it finishes). Both take the same steps:
+
+```python
+options = {"overwrite": "yes", "published": 1, "access_policy": "licensed", "repositoryid": "MY_COLLECTION"}
+
+# now: {"results": {...}, "errors": {...}} per step
+me.publish.project(123, catalog_connection_id=1, publish_thumbnail=False, options=options)
+
+# background job
+job = me.jobs.queue_publish_to_nada(123, catalog_connection_id=1, publish_thumbnail=False, options=options)
+result = me.jobs.wait(job["uuid"], timeout_seconds=3600)
+```
+
+Single steps: `me.publish.metadata`, `indicator_data`, `thumbnail`, `resources`, `resource`, `delete_nada_resources`.
+
+Step-by-step notebook: [`demo/publish_to_nada_jobs.ipynb`](demo/publish_to_nada_jobs.ipynb).
+
+## Guides
+
+- [Indicators](docs/indicators.md) — data structures, loading data, notes, publishing to NADA, pandas helpers
+- [Metadata workflows](docs/metadata-workflows.md) — outlines, [Pydantic](docs/metadata-workflows.md#pydantic), [Excel](docs/metadata-workflows.md#excel), [retrieving metadata](docs/metadata-workflows.md#retrieving-existing-metadata)
+- [LLM-assisted metadata](docs/llm-assisted-metadata.md) — draft and augment from documents and URLs
+- [API examples](docs/examples.md) — collections, copying projects, and common tasks
+- [Migrating from v1](legacy-migration.MD) — old → new method mapping
+- [API reference](docs/API_Reference.md) — regenerate with `python make_docs.py`
+
+## Contributing
+
+### Poetry troubleshooting (Windows)
+
+If NumPy or path-length errors occur:
+
+1. Enable [long paths in Windows](https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=powershell#enable-long-paths-in-windows-10-version-1607-and-later), or
+2. Run `poetry config virtualenvs.in-project true` then `poetry install`
+
+### Tests
+
+```bash
+poetry run pytest tests/test_interface.py -q
+```
+
+### API documentation
+
+```bash
+python make_docs.py             # demo notebooks -> docs/*.md, and docs/API_Reference.md
+python make_docs.py --api-only  # only docs/API_Reference.md
+```
+
+## Notes
+
+Parts of this codebase were written with the assistance of ChatGPT, in keeping with World Bank Group practice.
+

pymetadataeditor-0.5.0/README.md

@@ -0,0 +1,186 @@
+# pyMetadataEditor
+
+A Python client for [Metadata Editor](https://github.com/worldbank/metadata-editor): create, edit, and manage metadata for microdata, indicators, geospatial data, documents, scripts, images, and videos.
+
+## Installation
+
+This project uses [Poetry](https://python-poetry.org/docs/basic-usage/):
+
+```bash
+pip install poetry
+cd pymetadataeditor
+poetry install
+```
+
+The LLM-assisted metadata features (OpenAI, `tiktoken`, `markitdown`) are optional: `poetry install --extras llm`, or `pip install "pymetadataeditor[llm]"`.
+
+For development (pytest, pre-commit):
+
+```bash
+poetry install --with dev
+poetry run pre-commit install
+```
+
+## Quick start
+
+```python
+from pymetadataeditor import MetadataEditor
+import os
+
+me = MetadataEditor(
+    api_url=os.getenv("METADATA_API_URL"),   # e.g. https://host/index.php/api
+    api_key=os.getenv("METADATA_API_KEY"),
+    verify_ssl=False,  # set True when SSL is verified on your server
+)
+```
+
+### List projects
+
+`list` returns one page (summary fields, no metadata) plus the server `total`:
+
+```python
+response = me.projects.list(limit=10, metadata_type="indicator", keywords="literacy rate")
+print(response["total"])
+for project in response["projects"]:
+    print(project["id"], project["type"], project["title"])
+```
+
+Filters: `keywords`, `metadata_type`, `collection` (+ `exclude_collections=True`), `tag`, `ownership`, `users`,
+`created_by`, `changed_by`, `date_start` / `date_end` (last changed), `sort_by`. List filters accept Python lists.
+`me.projects.facets()` returns the available types, collections and their counts.
+
+To go through every matching project, iterate instead of loading everything at once:
+
+```python
+for project in me.projects.iter_all(metadata_type="indicator", page_size=500):
+    ...
+
+me.projects.count(collection=12)                # just the number
+me.projects.list_all(metadata_type="table")     # everything in memory: {"projects": [...], "total": n}
+```
+
+`iter_all` pages in id order, so projects edited while you iterate are neither skipped nor repeated. Other large
+listings have the same pattern: `me.collections.iter_projects`, `me.jobs.iter_all`, `me.codelists.iter_all`,
+`me.codelists.iter_codes`, `me.data_structures.iter_all`, `me.indicators.iter_data_rows`.
+
+### Create a project (template-aware)
+
+```python
+demo_name = "my_indicator_demo"
+metadata = {
+    "idno": demo_name,
+    "series_description": {
+        "idno": demo_name,
+        "name": "Version 1",
+    },
+}
+
+project_id = me.metadata.create(
+    metadata,
+    metadata_type_or_template_uid="indicator",
+)
+```
+
+### Read metadata back
+
+```python
+model = me.metadata.get(project_id, output_mode="pydantic")
+model.pretty_print()
+```
+
+## API overview (v2)
+
+Thin clients return **raw API JSON** (`dict`). Workflow helpers on `me.metadata` handle validation and Pydantic/Excel conversion.
+
+| Client | Role |
+|--------|------|
+| `me.projects` | List, get, create, update, patch, delete projects |
+| `me.metadata` | Template-aware create, update, get (dict / pydantic / excel) |
+| `me.collections` | Collections and membership |
+| `me.templates` | Template CRUD |
+| `me.resources` | External resources (reports, questionnaires, files) |
+| `me.jobs` | Background jobs (publish to NADA, indicator data import, poll, list) |
+| `me.publish` | Publish metadata and indicator DSD/data to NADA (synchronous) |
+| `me.uploads` | Resumable (chunked) file uploads |
+| `me.data_structures` | Data structure (DSD) registry: structures, components, CSV/JSON/SDMX import, export |
+| `me.codelists` | Codelist registry: codelists, codes, translations, import/export |
+| `me.indicators` | Indicator projects: bind a DSD, load/export data, chart data, `notes`, publish |
+| `me.api` | Generic `get` / `post` / `put` / `delete` for any path |
+
+API errors raise `ApiError` (`status_code`, `message`, `errors`), a subclass of `requests` `HTTPError`.
+
+Methods still on `me` directly include `make_metadata_outline`, `read_metadata_from_excel`, `draft_metadata_from_files`, and related helpers.
+
+**Breaking change:** v1 flat methods (`list_projects`, `create_project_log`, …) were removed. See [legacy-migration.MD](legacy-migration.MD).
+
+### Direct HTTP (`me.api`)
+
+For endpoints without a typed wrapper yet:
+
+```python
+me.api.get("/editor", params={"limit": 5})
+me.api.post("/collections/add_projects", json={"collections": [1], "id_format": "id", "projects": [101]})
+```
+
+POST body styles:
+
+- `json={...}` — JSON API payloads
+- `data={...}` — form fields
+- `data={...}` + `files={...}` — multipart uploads
+
+Many deletes use **POST** in restricted networks (e.g. `me.projects.delete`, `me.collections.delete`), not HTTP DELETE.
+
+### Publish to NADA
+
+`me.publish.*` publishes now and returns NADA's answers; `me.jobs.queue_*` methods queue a job on the
+editor's worker and return its `uuid` (pass `wait=True` to block until it finishes). Both take the same steps:
+
+```python
+options = {"overwrite": "yes", "published": 1, "access_policy": "licensed", "repositoryid": "MY_COLLECTION"}
+
+# now: {"results": {...}, "errors": {...}} per step
+me.publish.project(123, catalog_connection_id=1, publish_thumbnail=False, options=options)
+
+# background job
+job = me.jobs.queue_publish_to_nada(123, catalog_connection_id=1, publish_thumbnail=False, options=options)
+result = me.jobs.wait(job["uuid"], timeout_seconds=3600)
+```
+
+Single steps: `me.publish.metadata`, `indicator_data`, `thumbnail`, `resources`, `resource`, `delete_nada_resources`.
+
+Step-by-step notebook: [`demo/publish_to_nada_jobs.ipynb`](demo/publish_to_nada_jobs.ipynb).
+
+## Guides
+
+- [Indicators](docs/indicators.md) — data structures, loading data, notes, publishing to NADA, pandas helpers
+- [Metadata workflows](docs/metadata-workflows.md) — outlines, [Pydantic](docs/metadata-workflows.md#pydantic), [Excel](docs/metadata-workflows.md#excel), [retrieving metadata](docs/metadata-workflows.md#retrieving-existing-metadata)
+- [LLM-assisted metadata](docs/llm-assisted-metadata.md) — draft and augment from documents and URLs
+- [API examples](docs/examples.md) — collections, copying projects, and common tasks
+- [Migrating from v1](legacy-migration.MD) — old → new method mapping
+- [API reference](docs/API_Reference.md) — regenerate with `python make_docs.py`
+
+## Contributing
+
+### Poetry troubleshooting (Windows)
+
+If NumPy or path-length errors occur:
+
+1. Enable [long paths in Windows](https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=powershell#enable-long-paths-in-windows-10-version-1607-and-later), or
+2. Run `poetry config virtualenvs.in-project true` then `poetry install`
+
+### Tests
+
+```bash
+poetry run pytest tests/test_interface.py -q
+```
+
+### API documentation
+
+```bash
+python make_docs.py             # demo notebooks -> docs/*.md, and docs/API_Reference.md
+python make_docs.py --api-only  # only docs/API_Reference.md
+```
+
+## Notes
+
+Parts of this codebase were written with the assistance of ChatGPT, in keeping with World Bank Group practice.

pymetadataeditor-0.5.0/pymetadataeditor/__init__.py

@@ -0,0 +1,35 @@
+from .api_client import ApiClient
+from .codelists import CodelistsClient
+from .collections import CollectionsClient
+from .data_structures import DataStructuresClient, components_from_csv
+from .exceptions import ApiError, DeleteNotAppliedError, TemplateError
+from .indicators import IndicatorNotesClient, IndicatorsClient
+from .interface import MetadataEditor
+from .jobs import JobsClient
+from .metadata_workflows import MetadataWorkflowsClient
+from .projects import ProjectsClient
+from .publish import PublishClient
+from .resources import ResourcesClient
+from .templates_client import TemplatesClient
+from .uploads import UploadsClient
+
+__all__ = [
+    "MetadataEditor",
+    "ApiClient",
+    "ProjectsClient",
+    "CollectionsClient",
+    "TemplatesClient",
+    "ResourcesClient",
+    "PublishClient",
+    "JobsClient",
+    "UploadsClient",
+    "DataStructuresClient",
+    "CodelistsClient",
+    "IndicatorsClient",
+    "IndicatorNotesClient",
+    "components_from_csv",
+    "MetadataWorkflowsClient",
+    "ApiError",
+    "DeleteNotAppliedError",
+    "TemplateError",
+]

pymetadataeditor-0.5.0/pymetadataeditor/api_client.py

@@ -0,0 +1,76 @@
+"""Generic HTTP verb wrappers for calling any Metadata Editor API path."""
+
+from io import BufferedReader
+from typing import Dict, List, Optional, Union
+
+from .requester import RequestsWithSpecificErrors
+
+
+class ApiClient:
+    """Call API endpoints directly when no typed sub-client method exists yet."""
+
+    def __init__(self, api: RequestsWithSpecificErrors):
+        self._api = api
+
+    def request(
+        self,
+        method: str,
+        path: str,
+        id: Optional[Union[int, str]] = None,
+        params: Optional[Dict[str, Union[str, List[str]]]] = None,
+        json: Optional[Dict] = None,
+        data: Optional[Dict[str, str]] = None,
+        files: Optional[Dict[str, BufferedReader]] = None,
+    ) -> Dict:
+        """Issue an HTTP request against a path relative to the configured API base URL."""
+        return self._api.request(
+            method=method,
+            pth=path,
+            id=id,
+            params=params,
+            json=json,
+            data=data,
+            files=files,
+        )
+
+    def get(
+        self,
+        path: str,
+        id: Optional[Union[int, str]] = None,
+        params: Optional[Dict[str, Union[str, List[str]]]] = None,
+    ) -> Dict:
+        """GET request."""
+        return self.request("get", path, id=id, params=params)
+
+    def post(
+        self,
+        path: str,
+        id: Optional[Union[int, str]] = None,
+        json: Optional[Dict] = None,
+        data: Optional[Dict[str, str]] = None,
+        files: Optional[Dict[str, BufferedReader]] = None,
+        params: Optional[Dict[str, Union[str, List[str]]]] = None,
+    ) -> Dict:
+        """POST request. Prefer POST delete routes when DELETE is blocked by your network."""
+        return self.request("post", path, id=id, params=params, json=json, data=data, files=files)
+
+    def put(
+        self,
+        path: str,
+        id: Optional[Union[int, str]] = None,
+        json: Optional[Dict] = None,
+        data: Optional[Dict[str, str]] = None,
+        files: Optional[Dict[str, BufferedReader]] = None,
+        params: Optional[Dict[str, Union[str, List[str]]]] = None,
+    ) -> Dict:
+        """PUT request."""
+        return self.request("put", path, id=id, params=params, json=json, data=data, files=files)
+
+    def delete(
+        self,
+        path: str,
+        id: Optional[Union[int, str]] = None,
+        params: Optional[Dict[str, Union[str, List[str]]]] = None,
+    ) -> Dict:
+        """DELETE request. Many editor deletes use POST instead (e.g. `/editor/delete/{{id}}`)."""
+        return self.request("delete", path, id=id, params=params)

pymetadataeditor-0.5.0/pymetadataeditor/api_types.py

@@ -0,0 +1,21 @@
+"""Shared argument types and project type normalization for Metadata Editor API paths."""
+
+from typing import Dict, Union
+
+#: A project argument: the numeric project id (e.g. ``1234``) or the project idno (e.g. ``"ALB_2012_LSMS"``).
+#: IDNOs cannot be numeric only, so a numeric value is always the project id.
+ProjectIdOrIdno = Union[int, str]
+
+# Legacy / display names -> OpenAPI path segment (hyphenated).
+_PROJECT_TYPE_ALIASES: Dict[str, str] = {
+    "indicator": "timeseries",
+    "indicators_db": "timeseries-db",
+    "microdata": "survey",
+}
+
+
+def normalize_project_type(project_type: str) -> str:
+    """Map legacy or display type names to OpenAPI project-type path segments."""
+    key = project_type.strip().lower().replace("-", "_")
+    mapped = _PROJECT_TYPE_ALIASES.get(key, key)
+    return mapped.replace("_", "-")