gn2pg_client 1.8.0__tar.gz → 1.9.0.dev0__tar.gz

{gn2pg_client-1.8.0 → gn2pg_client-1.9.0.dev0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: gn2pg_client
-Version: 1.8.0
+Version: 1.9.0.dev0
 Summary: Import tool from GeoNature to a PostgreSQL database through Export module API (client side)
 License: AGPL-3.0-or-later
 Keywords: GeoNature,Export,SINP,opendata,biodiversity
@@ -58,11 +58,11 @@ Widely inspired from [ClientApiVN](https://framagit.org/lpo/Client_API_VN/)
 > [!WARNING]
 > The minimum version of the source GeoNature instance required for the incremental update must be a version 2.12.0
 
-![Project logo](./docs/source/_static/src_gn2pg.png)
+![Project logo](./docs/_static/src_gn2pg.png)
 
 ## Project Setup
 
-GN2PG Client can be installed by running `pip`. It requires Python 3.7.4 or above to run.
+GN2PG Client can be installed by running `pip`. It requires Python 3.9 or above to run.
 
 ```bash
 pip install gn2pg-client
@@ -85,7 +85,7 @@ gn2pg_cli --help
 This command init a TOML config file within `~/.gn2pg` hidden directory (in user `HOME` directory), named as you want. PLEASE DO NOT SPECIFY PATH!
 
 ```bash
-gn2pg_cli --init <myconfigfile>
+gn2pg_cli config --init <myconfigfile>
 ```
 
 Config file is structured as this. `[[source]]` block can be duplicate as many as needed (one block for each source).
@@ -149,7 +149,7 @@ data_type = "synthese_with_cd_nomenclature"
 To create json tables where datas will be stored, run :
 
 ```bash
-gn2pg_cli --json-tables-create <myconfigfile>
+gn2pg_cli db --json-tables-create <myconfigfile>
 ```
 
 ### Full download
@@ -157,7 +157,7 @@ gn2pg_cli --json-tables-create <myconfigfile>
 To download all datas from API, run :
 
 ```bash
-gn2pg_cli --full <myconfigfile>
+gn2pg_cli download --full <myconfigfile>
 ```
 
 ### Incremental download
@@ -165,13 +165,13 @@ gn2pg_cli --full <myconfigfile>
 To update data since last download, run :
 
 ```bash
-gn2pg_cli --update <myconfigfile>
+gn2pg_cli download --update <myconfigfile>
 ```
 
 To automate the launching of updates, you can write the cron task using the following command, for example every 30 minutes.
 
 ```cron
-*/30 * * * * /usr/bin/env bash -c "source <path to python environment>/bin/activate && gn2pg_cli --update <myconfigfile>" > /dev/null 2>&1
+*/30 * * * * /usr/bin/env bash -c "source <path to python environment>/bin/activate && gn2pg_cli download --update <myconfigfile>" > /dev/null 2>&1
 ```
 
 ### Debug mode
@@ -187,7 +187,7 @@ Log files are stored in `$HOME/.gn2pg/log` directory.
 Default script to auto populate GeoNature is called "to_gnsynthese".
 
 ```bash
-gn2pg_cli --custom-script to_gnsynthese <myconfigfile>
+gn2pg_cli db --custom-script to_gnsynthese <myconfigfile>
 ```
 
 > [!TIP]
@@ -228,16 +228,19 @@ poetry run gn2pg_cli <options>
 
 ## Team
 
-- [@lpofredc](https://github.com/lpofredc/) ([LPO Auvergne-Rhône-Alpes](https://github.com/lpoaura/)), main developer
+<a href="">
+<img height="100px" src="https://auvergne-rhone-alpes.lpo.fr/wp-content/uploads/LPO_AuRA.svg" title="DREAL AuRA">
+</a>
 
-![Logo LPOAuRA](https://raw.githubusercontent.com/lpoaura/biodivsport-widget/master/images/LPO_AuRA_l250px.png)
+[@lpofredc](https://github.com/lpofredc/) ([LPO Auvergne-Rhône-Alpes](https://github.com/lpoaura/)), lead developer
 
-- @ophdlv (Natural Solution), contributor
-- @mvergez (Natural Solution), contributor
-- @andriacap (Natural Solution), contributor
-- @Adrien-Pajot (Natural Solution), contributor
+---
 
-______________________________________________________________________
+<a href="https://github.com/lpoaura/GN2PG/graphs/contributors">
+  <img src="https://contrib.rocks/image?repo=lpoaura/GN2PG" />
+</a>
+
+---
 
 With the financial support of the [DREAL Auvergne-Rhône-Alpes](http://www.auvergne-rhone-alpes.developpement-durable.gouv.fr/) and the [Office français de la biodiversité](https://www.ofb.gouv.fr/).
 

{gn2pg_client-1.8.0 → gn2pg_client-1.9.0.dev0}/README.md

@@ -12,11 +12,11 @@ Widely inspired from [ClientApiVN](https://framagit.org/lpo/Client_API_VN/)
 > [!WARNING]
 > The minimum version of the source GeoNature instance required for the incremental update must be a version 2.12.0
 
-![Project logo](./docs/source/_static/src_gn2pg.png)
+![Project logo](./docs/_static/src_gn2pg.png)
 
 ## Project Setup
 
-GN2PG Client can be installed by running `pip`. It requires Python 3.7.4 or above to run.
+GN2PG Client can be installed by running `pip`. It requires Python 3.9 or above to run.
 
 ```bash
 pip install gn2pg-client
@@ -39,7 +39,7 @@ gn2pg_cli --help
 This command init a TOML config file within `~/.gn2pg` hidden directory (in user `HOME` directory), named as you want. PLEASE DO NOT SPECIFY PATH!
 
 ```bash
-gn2pg_cli --init <myconfigfile>
+gn2pg_cli config --init <myconfigfile>
 ```
 
 Config file is structured as this. `[[source]]` block can be duplicate as many as needed (one block for each source).
@@ -103,7 +103,7 @@ data_type = "synthese_with_cd_nomenclature"
 To create json tables where datas will be stored, run :
 
 ```bash
-gn2pg_cli --json-tables-create <myconfigfile>
+gn2pg_cli db --json-tables-create <myconfigfile>
 ```
 
 ### Full download
@@ -111,7 +111,7 @@ gn2pg_cli --json-tables-create <myconfigfile>
 To download all datas from API, run :
 
 ```bash
-gn2pg_cli --full <myconfigfile>
+gn2pg_cli download --full <myconfigfile>
 ```
 
 ### Incremental download
@@ -119,13 +119,13 @@ gn2pg_cli --full <myconfigfile>
 To update data since last download, run :
 
 ```bash
-gn2pg_cli --update <myconfigfile>
+gn2pg_cli download --update <myconfigfile>
 ```
 
 To automate the launching of updates, you can write the cron task using the following command, for example every 30 minutes.
 
 ```cron
-*/30 * * * * /usr/bin/env bash -c "source <path to python environment>/bin/activate && gn2pg_cli --update <myconfigfile>" > /dev/null 2>&1
+*/30 * * * * /usr/bin/env bash -c "source <path to python environment>/bin/activate && gn2pg_cli download --update <myconfigfile>" > /dev/null 2>&1
 ```
 
 ### Debug mode
@@ -141,7 +141,7 @@ Log files are stored in `$HOME/.gn2pg/log` directory.
 Default script to auto populate GeoNature is called "to_gnsynthese".
 
 ```bash
-gn2pg_cli --custom-script to_gnsynthese <myconfigfile>
+gn2pg_cli db --custom-script to_gnsynthese <myconfigfile>
 ```
 
 > [!TIP]
@@ -182,16 +182,19 @@ poetry run gn2pg_cli <options>
 
 ## Team
 
-- [@lpofredc](https://github.com/lpofredc/) ([LPO Auvergne-Rhône-Alpes](https://github.com/lpoaura/)), main developer
+<a href="">
+<img height="100px" src="https://auvergne-rhone-alpes.lpo.fr/wp-content/uploads/LPO_AuRA.svg" title="DREAL AuRA">
+</a>
 
-![Logo LPOAuRA](https://raw.githubusercontent.com/lpoaura/biodivsport-widget/master/images/LPO_AuRA_l250px.png)
+[@lpofredc](https://github.com/lpofredc/) ([LPO Auvergne-Rhône-Alpes](https://github.com/lpoaura/)), lead developer
 
-- @ophdlv (Natural Solution), contributor
-- @mvergez (Natural Solution), contributor
-- @andriacap (Natural Solution), contributor
-- @Adrien-Pajot (Natural Solution), contributor
+---
 
-______________________________________________________________________
+<a href="https://github.com/lpoaura/GN2PG/graphs/contributors">
+  <img src="https://contrib.rocks/image?repo=lpoaura/GN2PG" />
+</a>
+
+---
 
 With the financial support of the [DREAL Auvergne-Rhône-Alpes](http://www.auvergne-rhone-alpes.developpement-durable.gouv.fr/) and the [Office français de la biodiversité](https://www.ofb.gouv.fr/).
 

{gn2pg_client-1.8.0 → gn2pg_client-1.9.0.dev0}/gn2pg/api.py

@@ -166,13 +166,16 @@ class BaseAPI:
         self,
         params: dict,
         kind: str = "data",
-    ) -> Optional[List[str]]:
+        pagination_param: str = "offset",
+    ) -> tuple[Optional[List[str]], int]:
         """List offset pages to download data, based on API "total_filtered" and "limit" values
 
         :param params: Querystrings
         :type params: dict
         :param kind: kind of data, defaults to "data"
         :type kind: str, optional
+        :param pagination_param: Pagination parameter key
+        :type pagination_param: str, optional
         :return: url page list
         :rtype: Optional[List[str]]
         """
@@ -181,41 +184,44 @@ class BaseAPI:
 
         # Check kind value
         if self._url(kind) is None:
-            return None
+            return None, 0, None
 
         api_url = self._url(kind, params)
-
         try:
             response = session.get(url=api_url, params={**params})
+            status_code = response.status_code
+            if response.status_code == 200:
+                resp = response.json()
+                total_filtered = (
+                    resp["total_filtered"] if "total_filtered" in resp else resp["total"]
+                )
+                limit = resp["limit"]
+                total_pages = math.ceil(total_filtered / limit)
+                logger.debug(
+                    _("API %s contains %s data in %s page(s)"),
+                    api_url,
+                    total_filtered,
+                    total_pages,
+                )
+                if total_filtered > 0:
+                    page_list = list(
+                        self._url(
+                            kind,
+                            {
+                                **params,
+                                **{pagination_param: p + 1 if pagination_param == "page" else p},
+                            },
+                        )
+                        for p in range(total_pages)
+                    )
+                    return page_list, total_filtered, status_code
         except RetryError as e:
+            last_response = e.response
+            status_code = last_response.status_code
             logger.error(_("Export seems to be unavailable : %s"), e)
             raise RetryError from e
 
-        logger.debug(
-            _("Defining page_list from %s with status code %s"),
-            api_url,
-            response.status_code,
-        )
-        if response.status_code == 200:
-            resp = response.json()
-            total_filtered = resp["total_filtered"] if "total_filtered" in resp else resp["total"]
-            limit = resp["limit"]
-            total_pages = math.ceil(total_filtered / limit)
-            logger.debug(
-                _("API %s contains %s data in %s page(s)"),
-                api_url,
-                total_filtered,
-                total_pages,
-            )
-            if total_filtered > 0:
-                page_list = list(
-                    self._url(kind, {**params, **{"offset": p}}) for p in range(total_pages)
-                )
-                return page_list
-
-        logger.info(_("No data available from %s"), self._config.name)
-
-        return None
+        return None, 0, status_code
 
     def get_page(self, page_url: str) -> Optional[dict]:
         """Get data from one API page

gn2pg_client-1.9.0.dev0/gn2pg/app/admin_views.py

@@ -0,0 +1,58 @@
+"""Admin model views"""
+
+from flask_admin.contrib.sqla import ModelView
+
+from gn2pg.app.utils.admin_views import add_class_ctn_row_long_text_formatter, json_formatter
+
+
+class ReadOnlyView(ModelView):
+    """Generic read only admin model view"""
+
+    can_create = False
+    can_delete = False
+    can_edit = False
+
+
+class EditOnlyView(ReadOnlyView):
+    """Generic edit only admin model view"""
+
+    can_edit = True
+
+
+class ImportView(ReadOnlyView):
+    """Download log view"""
+
+    column_list = (
+        "id",
+        "source",
+        "controler",
+        "xfer_type",
+        "xfer_status",
+        "xfer_start_ts",
+        "xfer_end_ts",
+        "api_count_items",
+        "api_count_errors",
+        "data_count_upserts",
+        "data_count_delete",
+        "data_count_errors",
+        "metadata_count_upserts",
+        "metadata_count_errors",
+        "xfer_http_status",
+        "xfer_filters",
+        "comment",
+    )
+    column_default_sort = ("id", True)
+    column_filters = ("source", "xfer_type")
+    column_formatters = {
+        "xfer_filters": add_class_ctn_row_long_text_formatter,
+        "comment": add_class_ctn_row_long_text_formatter,
+    }
+    can_delete = True
+
+
+class ErrorView(ReadOnlyView):
+    """Error log view"""
+
+    column_list = ("source", "uuid", "controler", "last_ts", "item", "error", "import_id")
+    column_formatters = {"item": json_formatter, "error": add_class_ctn_row_long_text_formatter}
+    column_filters = ["source", "uuid", "controler", "import_id"]

{gn2pg_client-1.8.0 → gn2pg_client-1.9.0.dev0}/gn2pg/app/app.py

@@ -5,10 +5,10 @@ from flask_admin import Admin, AdminIndexView
 from werkzeug.middleware.proxy_fix import ProxyFix
 
 from gn2pg import _
-from gn2pg.app.admin_views import DownloadView, ErrorView, IncrementView
+from gn2pg.app.admin_views import ErrorView, ImportView
 from gn2pg.app.config import FlaskConfig
 from gn2pg.app.database import db
-from gn2pg.app.models import DownloadLog, ErrorLog, IncrementLog
+from gn2pg.app.models import ErrorLog, ImportLog
 
 
 def create_app(config=FlaskConfig):
@@ -18,6 +18,7 @@ def create_app(config=FlaskConfig):
     app.config.from_object(config)
     app.config["SQLALCHEMY_DATABASE_URI"] = config.sqlalchemy_database_uri
     app.config["APPLICATION_ROOT"] = config.application_root
+    app.config["SECRET_KEY"] = config.secret_key
     app.wsgi_app = ProxyFix(app.wsgi_app, x_host=1)
     db.init_app(app)
     app.config["db"] = db
@@ -30,8 +31,7 @@ def create_app(config=FlaskConfig):
         ),
         template_mode="bootstrap4",
     )
-    admin.add_view(DownloadView(DownloadLog, db.session, name=_("Full downloads")))
-    admin.add_view(IncrementView(IncrementLog, db.session, name=_("Incremental downloads")))
+    admin.add_view(ImportView(ImportLog, db.session, name=_("Imports")))
     admin.add_view(ErrorView(ErrorLog, db.session, name=_("Error logs")))
 
     return app

{gn2pg_client-1.8.0 → gn2pg_client-1.9.0.dev0}/gn2pg/app/config.py

@@ -48,6 +48,7 @@ DATABASES = {
 DEBUG = settings_config("DEBUG", default=False)
 ENV = settings_config("ENV", default="production")
 APPLICATION_ROOT = settings_config("APPLICATION_ROOT", default="/gn2pg")
+SECRET_KEY = settings_config("SECRET_KEY", default=None)
 
 
 class AppConfig:
@@ -62,6 +63,7 @@ class AppConfig:
     def __init__(self):
         self.env = ENV
         self.application_root = APPLICATION_ROOT
+        self.secret_key = SECRET_KEY
         self.debug = DEBUG
         self.database = DATABASES["default"]
         self.set_database_uri()

gn2pg_client-1.9.0.dev0/gn2pg/app/models.py

@@ -0,0 +1,69 @@
+"""Module providing Models according to database from config file."""
+
+from sqlalchemy.dialects.postgresql import JSONB, UUID
+from sqlalchemy.sql import func
+
+from gn2pg.app.database import db
+
+
+class ImportLog(db.Model):
+    """Download logs table"""
+
+    __tablename__ = "import_log"
+    __table_args__ = {"schema": "gn2pg_import"}
+    id = db.Column(db.Integer, db.Sequence("import_id_seq"), primary_key=True)
+    source = db.Column(db.String, nullable=False, index=True)
+    controler = db.Column(
+        db.String,
+        nullable=False,
+    )
+    xfer_type = db.Column(db.String, index=True)
+    xfer_status = db.Column(db.String, index=True)
+    xfer_http_status = db.Column(db.String, index=True)
+    xfer_filters = db.Column(JSONB, server_default="{}")
+    xfer_start_ts = db.Column(
+        db.DateTime(timezone=True),
+        nullable=False,
+    )
+    xfer_end_ts = db.Column(
+        db.DateTime(timezone=True),
+        nullable=True,
+    )
+    # api_count_data = db.Column(db.Integer, index=True, nullable=False)
+    api_count_errors = db.Column(db.Integer, nullable=False)
+    data_count_upserts = db.Column(db.Integer, index=True, nullable=False)
+    data_count_delete = db.Column(db.Integer, index=True, nullable=False)
+    data_count_errors = db.Column(db.Integer, index=True)
+    metadata_count_upserts = db.Column(db.Integer, index=True)
+    metadata_count_errors = db.Column(db.Integer, index=True)
+    comment = db.Column(db.String)
+
+    def __repr__(self):
+        return f"<Import_Log {self.id} {self.name}>"
+
+
+class ErrorLog(db.Model):
+    """Error logs table"""
+
+    __tablename__ = "error_log"
+    __table_args__ = {"schema": "gn2pg_import"}
+    source = db.Column(db.String, nullable=False, index=True, primary_key=True)
+    uuid = db.Column(UUID, nullable=False, index=True)
+    controler = db.Column(
+        db.String,
+        nullable=False,
+    )
+    last_ts = db.Column(
+        db.DateTime(timezone=True),
+        server_default=func.now(),
+        nullable=False,
+    )
+    item = db.Column(JSONB)
+    error = db.Column(db.String)
+    import_id = db.Column(
+        db.Integer,
+        db.ForeignKey("gn2pg_import.import_log.id", ondelete="CASCADE", onupdate="CASCADE"),
+    )
+
+    def __repr__(self):
+        return f"<error_log {self.name}>"