ckanapi-harvesters 0.0.0__tar.gz → 0.0.2__tar.gz

{ckanapi_harvesters-0.0.0 → ckanapi_harvesters-0.0.2}/LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2024 ifpen
+Copyright (c) 2026 IFPEN
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

{ckanapi_harvesters-0.0.0/src/ckanapi_harvesters.egg-info → ckanapi_harvesters-0.0.2}/PKG-INFO

@@ -1,10 +1,11 @@
 Metadata-Version: 2.4
 Name: ckanapi_harvesters
-Version: 0.0.0
-Summary: Adaptation of the template for github.
+Version: 0.0.2
+Summary: Package helping to upload local resources to a CKAN server using the CKAN API. Metadata and the list of resources can be defined in an Excel spreadsheet. The package includes requests to download resources and checks metadata against formatting rules.
+Maintainer-email: ifpen-gp <63413841+ifpen-gp@users.noreply.github.com>
 License: MIT License
         
-        Copyright (c) 2024 ifpen
+        Copyright (c) 2026 IFPEN
         
         Permission is hereby granted, free of charge, to any person obtaining a copy
         of this software and associated documentation files (the "Software"), to deal
@@ -24,25 +25,82 @@ License: MIT License
         OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
         SOFTWARE.
         
-Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.10
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Operating System :: OS Independent
-Requires-Python: >=3.12
+Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 Requires-Dist: pytest==8.0.1
+Requires-Dist: myst-nb>=1.3.0
+Requires-Dist: requests>=2.32.5
+Requires-Dist: pandas>=2.3.3
+Requires-Dist: numpy>=2.2.6
+Requires-Dist: openpyxl>=3.1.5
+Provides-Extra: harvesters
+Requires-Dist: pymongo>=4.8.0; extra == "harvesters"
+Requires-Dist: psycopg2>=2.9.11; extra == "harvesters"
+Requires-Dist: shapely>=2.1.2; extra == "harvesters"
+Requires-Dist: pyproj==3.6.1; extra == "harvesters"
+Provides-Extra: alt
+Requires-Dist: bson; extra == "alt"
+Requires-Dist: shapely>=2.1.2; extra == "alt"
+Requires-Dist: pyproj==3.6.1; extra == "alt"
+Requires-Dist: geopandas>=1.1.2; extra == "alt"
 
 # ckanapi_harvesters
 
+<img src="doc/assets/France2030-Logo-1024x576.png" alt="logo">
+
 ---
 
 ## Description
 
-This "Python Package" template is a complete template designed to create a Python package according to IFPEN's development standards, deployable internally within IFPEN or on the Cloud. This template provides developers with a Python project architecture in which they can contribute, document, and make it available to all IFPEN developers.
+This package enables users to benefit from the CKAN API and provides functions which
+realize complex API calls to achieve specific operations.
+In this package, DataStores are returned/inputted as pandas DataFrames.
+The underlying request mechanism uses the requests Session object, which improves performance with multiple requests.
+This package is oriented in the management of CKAN datasets and resources. 
+Only a selection of API calls has been implemented in this objective.
+To perform custom API calls, the function `api_action_call` is provided to the end user.
+This package was initially designed to harvest a large DataStores from your local file system.
+It also implements particular requests which can define a large DataStore.
+Large datasets composed of multiple files can be uploaded/downloaded 
+through scripts into a single resource or multiple resources. 
+For a DataStore, large files are uploaded with a limited number of rows per request.  
+
+The package is divided in the following sections:
+- `ckan_api`: functions interacting with the CKAN API. 
+  In addition to the base class which manages basic parameters and requests, API functions are divided as follows:
+  1) functions to map the CKAN packages and resources. The remote data structures are mapped in a mirrored data structure.
+     CKAN DataStore information, organizations, licenses and resource views are optionally tracked.
+  2) functions to query a DataStore or to download file resources.
+  3) functions to apply a test a data format policy on a given package.
+  4) functions to upsert data to a DataStore or to upload files to a resource.
+  5) functions to manage CKAN objects 
+     (creating, patching, or removing packages, resources, and DataStores).
+     These functions enable the user to change the metadata for these objects.
+     The other objects are meant to be managed through the API. 
+- `policies`: functions to check data format policies. A data format policy defines which attributes 
+  are mandatory for a package or resource. 
+  Specific rules can be implemented to restrict   package tags to certain lists, 
+  grouped by [vocabulary](https://docs.ckan.org/en/2.9/maintaining/tag-vocabularies.html).
+  Extra key-pair values of packages can be enforced. Resource formats can be restricted to a certain list.
+- `reports`: functions to extract a report on the CKAN database in order to monitor 
+  package user access rights, resource memory occupation, modification dates and data format policy messages. 
+- `harvesters`: this module implements ways to load data from your local machine. 
+  - `file_formats`: The primary approach is to use files on you local file system. The CSV and SHP (shape file) formats are currently supported. 
+  - In addition to the file formats, harvesters have been implemented to transfer data from a database. 
+  This is particularly useful if the database cannot be accessed by CKAN harvester extensions 
+  because it would only be available locally. MongoDB and PostgreSQL databases are currently supported.
+- `builder`: functions to automate package and resource metadata patching and data uploads or downloads.
+  These parameters can be defined in an Excel workbook and files from the local file system can be referred as inputs for the data upload.
+  The parameters can also be deduced from an online CKAN package through the API.
+  - Example scripts are given in this module, referring to an example Excel workbook.
+    The Excel workbook is available in the package and at this link:
+    [builder_package_example.xlsx](src/ckanapi_harvesters/builder/builder_package_example.xlsx)
+    See also the notebook example in the current documentation here: 
+    [builder_example_notebook.ipynb](sphinx/notebooks/builder_example_notebook.ipynb).
 
-This package includes:
-- A unit test structure based on the [Pytest](https://docs.pytest.org/en/stable/) library
-- Automatic documentation generation based on the [Sphinx](https://www.sphinx-doc.org/en/master/) library
-- A CI/CD pipeline for deploying the Python package to a Python server
 
 ## Github Pages
 
@@ -130,51 +188,29 @@ For more details on contributing and best practices, please refer to the `CONTRI
 
 ### Installation
 
-If you are using the provided `pip.conf`, you can simply run:
-
-```bash
-pip install ckanapi_harvesters
-```
-
-Otherwise, you can specify the package index depending on whether you are in an internal (on-premise) or external (cloud) environment.
+The package and its optional dependencies can be installed with the following command:
 
 ```bash
-# On-premise
-pip install ckanapi_harvesters --extra-index-url https://nexus.ifpen.fr/repository/fast-it/simple
-
-# On Cloud
-pip install ckanapi_harvesters --extra-index-url https://nexus.fastit.dev/repository/fast-it/simple
+pip install ckanapi_harvesters[extras]
 ```
 
-Alternatively, you can set the package index URL as an environment variable:
-
-```bash
-# On-premise
-export PIP_EXTRA_INDEX_URL=https://nexus.fastit.dev/repository/fast-it/simple
-
-# On Cloud
-export PIP_EXTRA_INDEX_URL=https://nexus.ifpen.fr/repository/fast-it/simple
-```
 
 ### Example Usage of the Python Package in Your Code
 
 After installation, you can import and use your package and its functions in your Python code:
 
 ```python
-from package import hello_world
+from ckanapi_harvesters import CkanApi
 
-hello_world()
+ckan = CkanApi()
 ```
 
 To use sub-modules defined in the package:
 
 ```python
-from package.divider import divide
-
-a = 4.0
-b = 2.0
+from ckanapi_harvesters.ckan_api import CkanApi
 
-c = divide(4., 2.)
+ckan = CkanApi()
 ```
 
 These instructions will allow you to access the package and utilize its features effectively and in line with your development configuration.

{ckanapi_harvesters-0.0.0 → ckanapi_harvesters-0.0.2}/README.md

@@ -1,15 +1,57 @@
 # ckanapi_harvesters
 
+<img src="doc/assets/France2030-Logo-1024x576.png" alt="logo">
+
 ---
 
 ## Description
 
-This "Python Package" template is a complete template designed to create a Python package according to IFPEN's development standards, deployable internally within IFPEN or on the Cloud. This template provides developers with a Python project architecture in which they can contribute, document, and make it available to all IFPEN developers.
+This package enables users to benefit from the CKAN API and provides functions which
+realize complex API calls to achieve specific operations.
+In this package, DataStores are returned/inputted as pandas DataFrames.
+The underlying request mechanism uses the requests Session object, which improves performance with multiple requests.
+This package is oriented in the management of CKAN datasets and resources. 
+Only a selection of API calls has been implemented in this objective.
+To perform custom API calls, the function `api_action_call` is provided to the end user.
+This package was initially designed to harvest a large DataStores from your local file system.
+It also implements particular requests which can define a large DataStore.
+Large datasets composed of multiple files can be uploaded/downloaded 
+through scripts into a single resource or multiple resources. 
+For a DataStore, large files are uploaded with a limited number of rows per request.  
+
+The package is divided in the following sections:
+- `ckan_api`: functions interacting with the CKAN API. 
+  In addition to the base class which manages basic parameters and requests, API functions are divided as follows:
+  1) functions to map the CKAN packages and resources. The remote data structures are mapped in a mirrored data structure.
+     CKAN DataStore information, organizations, licenses and resource views are optionally tracked.
+  2) functions to query a DataStore or to download file resources.
+  3) functions to apply a test a data format policy on a given package.
+  4) functions to upsert data to a DataStore or to upload files to a resource.
+  5) functions to manage CKAN objects 
+     (creating, patching, or removing packages, resources, and DataStores).
+     These functions enable the user to change the metadata for these objects.
+     The other objects are meant to be managed through the API. 
+- `policies`: functions to check data format policies. A data format policy defines which attributes 
+  are mandatory for a package or resource. 
+  Specific rules can be implemented to restrict   package tags to certain lists, 
+  grouped by [vocabulary](https://docs.ckan.org/en/2.9/maintaining/tag-vocabularies.html).
+  Extra key-pair values of packages can be enforced. Resource formats can be restricted to a certain list.
+- `reports`: functions to extract a report on the CKAN database in order to monitor 
+  package user access rights, resource memory occupation, modification dates and data format policy messages. 
+- `harvesters`: this module implements ways to load data from your local machine. 
+  - `file_formats`: The primary approach is to use files on you local file system. The CSV and SHP (shape file) formats are currently supported. 
+  - In addition to the file formats, harvesters have been implemented to transfer data from a database. 
+  This is particularly useful if the database cannot be accessed by CKAN harvester extensions 
+  because it would only be available locally. MongoDB and PostgreSQL databases are currently supported.
+- `builder`: functions to automate package and resource metadata patching and data uploads or downloads.
+  These parameters can be defined in an Excel workbook and files from the local file system can be referred as inputs for the data upload.
+  The parameters can also be deduced from an online CKAN package through the API.
+  - Example scripts are given in this module, referring to an example Excel workbook.
+    The Excel workbook is available in the package and at this link:
+    [builder_package_example.xlsx](src/ckanapi_harvesters/builder/builder_package_example.xlsx)
+    See also the notebook example in the current documentation here: 
+    [builder_example_notebook.ipynb](sphinx/notebooks/builder_example_notebook.ipynb).
 
-This package includes:
-- A unit test structure based on the [Pytest](https://docs.pytest.org/en/stable/) library
-- Automatic documentation generation based on the [Sphinx](https://www.sphinx-doc.org/en/master/) library
-- A CI/CD pipeline for deploying the Python package to a Python server
 
 ## Github Pages
 
@@ -97,51 +139,29 @@ For more details on contributing and best practices, please refer to the `CONTRI
 
 ### Installation
 
-If you are using the provided `pip.conf`, you can simply run:
-
-```bash
-pip install ckanapi_harvesters
-```
-
-Otherwise, you can specify the package index depending on whether you are in an internal (on-premise) or external (cloud) environment.
+The package and its optional dependencies can be installed with the following command:
 
 ```bash
-# On-premise
-pip install ckanapi_harvesters --extra-index-url https://nexus.ifpen.fr/repository/fast-it/simple
-
-# On Cloud
-pip install ckanapi_harvesters --extra-index-url https://nexus.fastit.dev/repository/fast-it/simple
+pip install ckanapi_harvesters[extras]
 ```
 
-Alternatively, you can set the package index URL as an environment variable:
-
-```bash
-# On-premise
-export PIP_EXTRA_INDEX_URL=https://nexus.fastit.dev/repository/fast-it/simple
-
-# On Cloud
-export PIP_EXTRA_INDEX_URL=https://nexus.ifpen.fr/repository/fast-it/simple
-```
 
 ### Example Usage of the Python Package in Your Code
 
 After installation, you can import and use your package and its functions in your Python code:
 
 ```python
-from package import hello_world
+from ckanapi_harvesters import CkanApi
 
-hello_world()
+ckan = CkanApi()
 ```
 
 To use sub-modules defined in the package:
 
 ```python
-from package.divider import divide
-
-a = 4.0
-b = 2.0
+from ckanapi_harvesters.ckan_api import CkanApi
 
-c = divide(4., 2.)
+ckan = CkanApi()
 ```
 
 These instructions will allow you to access the package and utilize its features effectively and in line with your development configuration.

ckanapi_harvesters-0.0.2/pyproject.toml

@@ -0,0 +1,96 @@
+[project]
+name = "ckanapi_harvesters"
+version = '0.0.2'
+authors = [
+]
+maintainers = [
+	{ name = "ifpen-gp", email = "63413841+ifpen-gp@users.noreply.github.com" },
+]
+description = "Package helping to upload local resources to a CKAN server using the CKAN API. Metadata and the list of resources can be defined in an Excel spreadsheet. The package includes requests to download resources and checks metadata against formatting rules."
+readme = "README.md"
+requires-python = ">=3.10"
+classifiers = [
+    "Programming Language :: Python :: 3.10",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+]
+dependencies = [
+    "pytest == 8.0.1",
+    "myst-nb >= 1.3.0",
+    # ajouter les dépendances necessaires
+	"requests >= 2.32.5",
+	"pandas >= 2.3.3",
+	"numpy >= 2.2.6",
+	"openpyxl >= 3.1.5",  # could be an optional extra
+]
+
+license = { file = "LICENSE" }
+
+[project.optional-dependencies]
+harvesters = [
+    "pymongo >= 4.8.0",    # MongoDB harvester
+    "psycopg2 >= 2.9.11",  # Postgre harvester
+    "shapely >= 2.1.2",    # GeoJSON support
+    "pyproj == 3.6.1",     # GeoJSON transformations (v3.6.1 is compatible with Python 3.10)
+]
+alt = [
+    "bson",  # bson implementation, independent of pymongo
+    "shapely >= 2.1.2",    # GeoJSON support
+    "pyproj == 3.6.1",     # GeoJSON transformations (v3.6.1 is compatible with Python 3.10)
+    "geopandas >= 1.1.2",  # Shapefile reading functions
+]
+
+[tool.setuptools.package-data]
+# non-source files to include in the package
+"ckanapi_harvesters" = [
+    "builder/builder_package_example.xlsx",
+    "builder/example/package/*.*",
+    "builder/example/*.ipynb",
+]
+
+[tool.ckanapi_harvesters]
+version = "1.0.2"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools_scm]
+version_scheme = "guess-next-dev"
+local_scheme = "dirty-tag"
+
+[tool.setuptools]
+py-modules = []
+license-files = []
+
+[tool.ruff]
+# Set the maximum line length to 140.
+line-length = 140
+exclude = [
+    "tests",
+    "sphinx",
+    "src",
+]
+
+[tool.ruff.lint]
+# See documentation: https://docs.astral.sh/ruff/rules/
+extend-select = [
+  "UP",
+  "E501",
+  "I",
+  "B",
+  "F",
+  "E",
+  "N",
+  "A",
+  "PL",
+  "D"
+]
+
+
+[tool.ruff.lint.pydocstyle]
+convention = "numpy"  # needs to be changed (rst docstrings)
+
+[tool.ruff.format]
+quote-style = "single"
+indent-style = "space"
+docstring-code-format = true

ckanapi_harvesters-0.0.2/src/ckanapi_harvesters/__init__.py

@@ -0,0 +1,37 @@
+#!python3
+# -*- coding: utf-8 -*-
+"""
+Package with helper function for CKAN requests using pandas DataFrames.
+"""
+
+# builder_file_format_version = "0.0.1"
+try:
+    from importlib.metadata import version, PackageNotFoundError
+except ImportError:  # Python <3.8
+    from importlib_metadata import version, PackageNotFoundError
+
+try:
+    __version__ = version("ckanapi_harvesters")
+except PackageNotFoundError:
+    __version__ = None
+
+
+import os
+self_dir = os.path.abspath(os.path.dirname(os.path.realpath(__file__)))
+
+
+from . import auxiliary
+from . import policies
+from . import harvesters
+from . import ckan_api
+from . import builder
+from . import reports
+
+# usage shortcuts
+from .auxiliary import CkanMap
+from .policies import CkanPackageDataFormatPolicy
+from .ckan_api import CkanApi, CKAN_API_VERSION
+from .builder import BUILDER_FILE_FORMAT_VERSION
+from .builder import BuilderPackage, BuilderDataStoreMultiABC, BuilderDataStoreFolder, RequestFileMapperIndexKeys
+
+

ckanapi_harvesters-0.0.2/src/ckanapi_harvesters/auxiliary/__init__.py

@@ -0,0 +1,26 @@
+#!python3
+# -*- coding: utf-8 -*-
+"""
+Package with helper function for CKAN requests using pandas DataFrames.
+"""
+
+from . import ckan_defs
+from . import path
+from . import login
+from . import urls
+from . import proxy_config
+from . import external_code_import
+from . import list_records
+from . import ckan_action
+from . import ckan_errors
+from . import ckan_configuration
+from . import ckan_api_key
+from . import ckan_model
+from . import ckan_map
+from . import ckan_vocabulary_deprecated
+from . import ckan_auxiliary
+from . import deprecated
+
+from .ckan_map import CkanMap
+from .external_code_import import unlock_external_code_execution
+

ckanapi_harvesters-0.0.2/src/ckanapi_harvesters/auxiliary/ckan_action.py

@@ -0,0 +1,93 @@
+#!python3
+# -*- coding: utf-8 -*-
+"""
+Action response common treatments
+"""
+from typing import Union
+import json
+
+import requests
+
+
+class CkanActionResponse:
+    """
+    Class which decodes and checks the response of a CKAN request
+    """
+    def __init__(self, response: requests.Response, dry_run: bool=False):
+        self.response: requests.Response = response  # for debug purposes
+        self.response_dict:Union[dict,None] = None
+        self.status_code:int = response.status_code
+        self.success:bool = False
+        self.success_json_loads:bool = False
+        self.result:Union[dict,None] = None
+        self.error_message: Union[None,str,dict] = None
+        self.len:Union[int,None] = None
+        self.dry_run:bool = dry_run
+
+        if response.content is None and response.request is None:
+            # dry run
+            assert(dry_run)
+            self.success = True
+            self.success_json_loads = False
+            self.status_code = 1
+            self.error_message = "Request not sent: dry run mode"
+            self.len = 0
+        else:
+            try:
+                response_dict = json.loads(response.content.decode())
+                self.response_dict = response_dict
+                self.success_json_loads = True
+                if (response.status_code == 200 and "success" in response_dict.keys() and "result" in response_dict.keys()
+                        and response_dict["success"]):
+                    self.success = True
+                    self.result = response_dict["result"]
+                else:
+                    if "error" in response_dict.keys():
+                        self.error_message = response_dict["error"]
+                    else:
+                        self.error_message = response.content.decode()
+            except Exception as json_error:
+                self.error_message = f"JSON decode error {json_error} & CKAN error {response.content.decode()}"
+
+    def __len__(self):
+        if self.len is None:
+            raise RuntimeError("queried len but does not have len")
+        return self.len
+
+    def default_error(self, ckan) -> "CkanActionError":
+        """
+        Raise specific error codes depending on response
+        """
+        if self.status_code == 404 and self.success_json_loads and self.error_message["__type"] == "Not Found Error":
+            return CkanNotFoundError(ckan, "(Generic)", self)
+        elif self.status_code == 403 and self.success_json_loads and self.error_message["__type"] == "Authorization Error":
+            return CkanAuthorizationError(ckan, self)
+        else:
+            return CkanActionError(ckan, self)
+
+## action error codes
+class CkanActionError(Exception):
+    def __init__(self, ckan, response: CkanActionResponse, display_request:bool=True):
+        super().__init__(response.error_message)
+        self.response = response
+        self.status_code = response.status_code
+        if display_request:
+            ckan._error_print_debug_response(response.response)
+
+    def __str__(self):
+        return f"Server code [{self.status_code}]: " + super().__str__()
+
+class CkanNotFoundError(CkanActionError):
+    def __init__(self, ckan, object_type:str, response: CkanActionResponse, display_request:bool=True):
+        response.error_message = f"{object_type} not found: {response.error_message}"
+        super().__init__(ckan, response, display_request=display_request)
+        self.object_type = object_type
+
+class CkanAuthorizationError(CkanActionError):
+    pass
+
+class CkanSqlCapabilityError(CkanActionError):
+    def __init__(self, ckan, response: CkanActionResponse, display_request:bool=True):
+        response.error_message = f"sql capabilities are not activated on CKAN server. See documentation for option ckan.datastore.sqlsearch.enabled"
+        super().__init__(ckan, response, display_request=display_request)
+