openmodule 13.1.4__tar.gz → 13.3.0__tar.gz

{openmodule-13.1.4 → openmodule-13.3.0}/ChangeLog

@@ -1,6 +1,19 @@
 CHANGES
 =======
 
+v13.3.0
+-------
+
+* [OM-359] cleanup helper function
+
+v13.2.0
+-------
+
+* OM 187 Alembic migration test mixin
+* add a testcase for expiring session data while adding new models (noticed in OM-327)
+* Aktualisierung der Dokumentation
+* OM-255
+
 v13.1.4
 -------
 
@@ -145,27 +158,6 @@ v11.1.0.rc4
 
 * arivo-schedule in public pip
 * use temporary arivo-schedule package
-
-v11.1.0.rc3
------------
-
-* changed docs and requirements for schedule library until its a pip package
-
-v11.1.0.rc2
------------
-
-* added fork of schedule package
-
-v11.1.0.rc1
------------
-
-* databox upload now can have dst as folder (endswith "/")
-
-v11.1.0.rc0
------------
-
-* fixed testcase
-* small changes in csv export and databox upload adding more testcases docs
 * removed mock rpcs from schema
 
 v11.0.3
@@ -226,8 +218,3 @@ v10.0.1
 v10.0.0
 -------
 
-* v10
-* README Update for creation of dist structure [skip ci]
-* don't schedule any more rpc's if we are already shutdown
-* Update README.md
-* Update README.md

openmodule-13.3.0/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.1
+Name: openmodule
+Version: 13.3.0
+Summary: Libraries for developing the arivo openmodule
+Home-page: https://gitlab.com/arivo-public/device-python/openmodule.git
+Author: ARIVO
+Author-email: support@arivo.co
+License: GNU General Public License v2 (GPLv2)
+Keywords: arivo openmodule
+Platform: UNKNOWN
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: GNU General Public License v2 (GPLv2)
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
+Description-Content-Type: text/markdown; charset=UTF-8
+Provides-Extra: test
+Provides-Extra: commands
+License-File: LICENSE
+
+# OpenModule V2
+
+[TOC]
+
+## Quickstart
+
+You can install this library via pip:
+```bash
+pip install openmodule
+```
+
+#### Development with Feature Branches
+
+During development it might be necessary to install a version of openmodule, where no pip package exists.
+Below you can find how to install a certain openmodule branch for your application with pip.
+
+##### Openmodule
+
+Bash command:
+```bash
+pip install "git+https://gitlab.com/arivo-public/device-python/openmodule@<branch>#egg=openmodule"
+```
+
+requirements.txt:
+```text
+git+https://gitlab.com/arivo-public/device-python/openmodule@<branch>#egg=openmodule
+```
+
+##### Openmodule Test
+
+Bash command:
+```bash
+pip install "git+https://gitlab.com/arivo-public/device-python/openmodule@<branch>#egg=openmodule-test&subdirectory=openmodule_test"
+```
+
+requirements.txt:
+```text
+git+https://gitlab.com/arivo-public/device-python/openmodule@<branch>#egg=openmodule-test&subdirectory=openmodule_test
+```
+
+##### Openmodule Commands
+
+Bash command:
+```bash
+pip install "git+https://gitlab.com/arivo-public/device-python/openmodule@<branch>#egg=openmodule-commands&subdirectory=openmodule_commands
+```
+
+requirements.txt:
+```text
+git+https://gitlab.com/arivo-public/device-python/openmodule@<branch>#egg=openmodule-commands&subdirectory=openmodule_commands
+```
+
+#### Local Development
+
+Sometimes you want to test local changes of the Openmodule library in device services and therefore you can do a local
+installation of the library. We use the
+[editable installs](https://pip.pypa.io/en/stable/topics/local-project-installs/#editable-installs) of Pip for this.
+
+##### Openmodule
+
+bash command:
+```bash
+pip install -e <path_to_openmodule_root>
+```
+
+##### Openmodule Test
+
+bash command:
+```bash
+pip install -e <path_to_openmodule_root>/openmodule_test/
+```
+
+##### Openmodule Commands
+
+bash command:
+```bash
+pip install -e <path_to_openmodule_root>/openmodule_commands/
+```
+
+## Changes
+
+- [Breaking Changes](docs/migrations.md)
+- [Known Issues](docs/known_issues.md)
+
+## Documentation
+
+### Openmodule
+
+- [Getting Started](docs/getting_started.md)
+- [Coding Standard](docs/coding_standard.md)
+- [Settings](docs/settings.md)
+- [RPC](docs/rpc.md)
+- [Health](docs/health.md)
+- [Database](docs/database.md)
+- [Eventlog](docs/event_sending.md)
+- [Package Reader](docs/package_reader.md)
+- [Anonymization](docs/anonymization.md)
+- [Connection Status Listener](docs/connection_status_listener.md)
+- [Settings Provider](docs/settings_provider.md)
+- [Access Service](docs/access_service.md)
+- [CSV Export](docs/csv_export.md)
+- [Translations](docs/translation.md)
+- [Utils](docs/utils.md)
+- [Deprecated Features](docs/deprecated.md)
+- [Testing](docs/testing.md)
+- [File Cleanup](docs/cleanup.md)
+
+### Openmodule Commands
+
+- [Openmodule Commands](docs/commands.md)
+
+
+

openmodule-13.3.0/README.md

@@ -0,0 +1,110 @@
+# OpenModule V2
+
+[TOC]
+
+## Quickstart
+
+You can install this library via pip:
+```bash
+pip install openmodule
+```
+
+#### Development with Feature Branches
+
+During development it might be necessary to install a version of openmodule, where no pip package exists.
+Below you can find how to install a certain openmodule branch for your application with pip.
+
+##### Openmodule
+
+Bash command:
+```bash
+pip install "git+https://gitlab.com/arivo-public/device-python/openmodule@<branch>#egg=openmodule"
+```
+
+requirements.txt:
+```text
+git+https://gitlab.com/arivo-public/device-python/openmodule@<branch>#egg=openmodule
+```
+
+##### Openmodule Test
+
+Bash command:
+```bash
+pip install "git+https://gitlab.com/arivo-public/device-python/openmodule@<branch>#egg=openmodule-test&subdirectory=openmodule_test"
+```
+
+requirements.txt:
+```text
+git+https://gitlab.com/arivo-public/device-python/openmodule@<branch>#egg=openmodule-test&subdirectory=openmodule_test
+```
+
+##### Openmodule Commands
+
+Bash command:
+```bash
+pip install "git+https://gitlab.com/arivo-public/device-python/openmodule@<branch>#egg=openmodule-commands&subdirectory=openmodule_commands
+```
+
+requirements.txt:
+```text
+git+https://gitlab.com/arivo-public/device-python/openmodule@<branch>#egg=openmodule-commands&subdirectory=openmodule_commands
+```
+
+#### Local Development
+
+Sometimes you want to test local changes of the Openmodule library in device services and therefore you can do a local
+installation of the library. We use the
+[editable installs](https://pip.pypa.io/en/stable/topics/local-project-installs/#editable-installs) of Pip for this.
+
+##### Openmodule
+
+bash command:
+```bash
+pip install -e <path_to_openmodule_root>
+```
+
+##### Openmodule Test
+
+bash command:
+```bash
+pip install -e <path_to_openmodule_root>/openmodule_test/
+```
+
+##### Openmodule Commands
+
+bash command:
+```bash
+pip install -e <path_to_openmodule_root>/openmodule_commands/
+```
+
+## Changes
+
+- [Breaking Changes](docs/migrations.md)
+- [Known Issues](docs/known_issues.md)
+
+## Documentation
+
+### Openmodule
+
+- [Getting Started](docs/getting_started.md)
+- [Coding Standard](docs/coding_standard.md)
+- [Settings](docs/settings.md)
+- [RPC](docs/rpc.md)
+- [Health](docs/health.md)
+- [Database](docs/database.md)
+- [Eventlog](docs/event_sending.md)
+- [Package Reader](docs/package_reader.md)
+- [Anonymization](docs/anonymization.md)
+- [Connection Status Listener](docs/connection_status_listener.md)
+- [Settings Provider](docs/settings_provider.md)
+- [Access Service](docs/access_service.md)
+- [CSV Export](docs/csv_export.md)
+- [Translations](docs/translation.md)
+- [Utils](docs/utils.md)
+- [Deprecated Features](docs/deprecated.md)
+- [Testing](docs/testing.md)
+- [File Cleanup](docs/cleanup.md)
+
+### Openmodule Commands
+
+- [Openmodule Commands](docs/commands.md)

{openmodule-13.1.4 → openmodule-13.3.0}/docs/access_service.md

@@ -1,5 +1,7 @@
 # Access Service
 
+[TOC]
+
 Inherit from either
 
 * Access Service: No session handling and no access sync

openmodule-13.3.0/docs/anonymization.md

@@ -0,0 +1,52 @@
+### Anonymization
+
+[TOC]
+
+The openmodule framework uses rpc requests and messages to trigger the anonymization of data.
+
+* **Message:** ONLY the DSGVO device service can send a AnonymizeMessage (topic: `privacy`).  
+  The message includes a session_id and vehicle_ids to delete.
+* **RPC Request:** You can send an AnonymizeRequest with channel=`privacy`, type=`anonymize` to the DSGVO container.
+  This request only includes session_ids.
+  The DSGVO container will then match vehicle_ids to the session_ids and redistribute the request with the prior
+  mentioned message.
+
+A container with sensitive data then needs to implement the message listener for the privacy messages (see example).
+
+##### Example 1
+
+The controller checked that a parking session was finished and fully paid. After a specified time, the DSGVO relevant
+data has to be anonymized. The controller then triggers the anonymization.
+
+```python
+request = AnonymizeRequest(session_ids=[session_id])
+result = core.rpc_client.rpc("privacy", "anonymize", request)
+if result.response.status == "ok":
+    self.log.info(f"Anonymized session {session_id}")
+```
+
+##### Example 2
+
+The controller checked that a parking session was finished and fully paid. After a specified time, the DSGVO relevant
+data has to be anonymized. The controller then triggers the anonymization.
+
+```python
+msg = AnonymizeMessage(vehicle_ids=[vid1, vid2])
+self.core.publish(msg, "privacy")
+```
+
+The DSGVO container receives the request, matches session_ids with vehicle_ids and publishes the anonymization message.
+It also listens on said messages and deletes vehicle images based on the vehicle_ids in the message.
+
+```python
+core.messages.register("privacy", AnonymizeMessage, anonymize_data)
+
+
+def anonymize_data(message: AnonymizeMessage):
+    for vid in message.vehicle_ids:
+        delete_vehicle_image_by_vehicle_id(vid)
+```
+
+**IMPORTANT** You still have to take care of data retention in each service separately, meaning you have to delete data
+independently of these anonymization messages.
+i.e. the DSGVO service deletes data if we need disk space or the eventlog deletes events after 30 days by default

openmodule-13.3.0/docs/cleanup.md

@@ -0,0 +1,42 @@
+# File cleanup
+
+With the exception for some globally defined file_patterns (e.g. debug images, log rotation files) every service is responsible for the cleanup of its files.
+The helper function `schedule_file_cleanup` schedules a file cleanup at 2:xx local time, where xx is service name dependent to avoid that all services delete at the same time.
+Important notes:
+* only files will be deleted, directories are kept
+* files are only deleted once per day
+* files are deleted based on modified timestamp
+* a scheduler must be used which does `.run_pending()` regularly
+* as glob is only inclusive, an exclude regex can be specified for special cases
+
+
+## Examples
+
+```python
+import time
+import re
+import schedule
+from datetime import timedelta
+from openmodule.utils.cleanup import schedule_file_cleanup
+
+scheduler = schedule.Scheduler()
+
+# delete specific file after 30 days
+schedule_file_cleanup(scheduler, "/data/file_to_keep_30_days.txt", timedelta(days=30))
+
+# delete png files in folder after 1 day
+schedule_file_cleanup(scheduler, "/data/images/*.png", timedelta(days=1))
+
+# delete jpg files in subfolders after 30 days
+schedule_file_cleanup(scheduler, "/data/*/*.jpg", timedelta(days=30))
+
+# delete jpg files in subfolders recursively after 30 days (e.g. /data/a.jpg, /data/d1/a.jpg, /data/d1/d2/a.jpg)
+schedule_file_cleanup(scheduler, "/data/**/*.jpg", timedelta(days=30))
+
+# delete jpg files in subfolders, except if regex matches, recursively after 30 days (e.g. /data/d1/a.jpg, but not /data/vehicle_images/a.jpg)
+schedule_file_cleanup(scheduler, "/data/**/*.jpg", timedelta(days=30), re.compile('.*/vehicle_images/.*'))
+
+while True:
+    scheduler.run_pending()
+    time.sleep(1.0)
+```

openmodule-13.3.0/docs/coding_standard.md

@@ -0,0 +1,79 @@
+# Openmodule Coding Standard
+
+[TOC]
+
+## Models
+
+* All Messages or RPC requests MUST be defined as pydantic models in either `openmodule/models` or in `src/models` to
+  avoid confusion about datatypes.
+
+* All Models MUST be must inherit `OpenModuleModel` or one of its children
+
+* ZMQ Messages MUST inherit `ZMQMessage`
+
+### Enum
+
+As a coding standard we are using camel case and the name of the enum and value should always be the same.
+This means you are not allowed to use a dash (`-`) in the enum value.
+
+We are also using the `str` type for all enums, because we want to compare the enum value against strings.
+
+```python
+from enum import Enum
+
+
+class MyStatus(str, Enum):
+    very_good = "very_good"
+    good = "good"
+    this_is_fine = "this_is_fine"
+    something_is_wrong = "something_is_wrong"
+```
+
+## Database
+
+* All database models must be defined within either `openmodule/models` or in the directory `src/database`
+
+* All database model names have to end with Model, i.e. TestModel(Base)
+
+* All database interactions MUST be defined in functions under `src/database/database.py`. There are no other access of
+  the database in any other file.
+
+* All functions in `src/database/database.py` MUST have the database/session as first parameter: `def stuff(db, ...)`
+
+* All table names MUST use "_" as separator
+
+* For single object queries only use `query.one()` and `query.first()`
+
+## Typing
+
+We use type annotation for functions and variables.
+This not only improves read ability of the source code, but the IDEs and linters can
+also use this information to provide better code completion and error checking.
+
+### Examples
+
+```python
+class MyPriceModel(OpenModuleModel):
+    price: float  # net price
+    currency: str  # currency
+    vat: float = 0  # amount of vat for country
+
+
+vat_country_in_percent: float = 20.0
+my_products: Dict[str, MyPriceModel] = {"apple": MyPriceModel(price=1.0, currency="EUR")}
+
+
+def calculate_vat(price: MyPriceModel) -> float:
+    """Calculate the vat for a product."""
+    return price.price * vat_country_in_percent / 100.0
+```
+
+
+We allow that function that are not returning anything to not have a return type annotation.
+
+```python
+def updating_vat(products: Dict[str, MyPriceModel]):
+    """We are directly updating the vat in the products, therefore we have no return value."""
+    for product in products.values():
+        product.vat = product.price * vat_country_in_percent / 100.0
+```

{openmodule-13.1.4 → openmodule-13.3.0}/docs/commands.md

@@ -1,4 +1,7 @@
 # Openmodule Commands
+
+[TOC]
+
 We provide multiple commands that are helpful for developing openmodule services.
 
 ## Create new commands
@@ -12,6 +15,6 @@ Creating new commands is very simple:
 ## Translations
 
 Commands for creating translations, for more info see [here](translation.md)
-* **openmodule_makemessage:** Creates the translation files and opens a editor for translating
-* **openmodule_translate:** Opens a editor for translating
-* **openmodule_checktranslation:** Checks if the translations are up to date and if any translations are empty
+* **openmodule_makemessage:** Creates the translation files and opens an editor for translating
+* **openmodule_translate:** Opens an editor for translating
+* **openmodule_checktranslation:** Checks if the translations are up-to-date and if any translations are empty

{openmodule-13.1.4 → openmodule-13.3.0}/docs/connection_status_listener.md

@@ -1,5 +1,7 @@
 # ConnectionStatusListener
 
+[TOC]
+
 For some functionality we need to know if we have an active connection to the server or in the case of bridged installations if we have a connection to the main compute unit.
 For this you can use the ConnectionStatusListener, which will be automatically created in the core.
 
@@ -11,7 +13,7 @@ The connection to the server has 4 different states:
 * **offline:** The connection is inactive for some time
 * **startup:** We just left the offline state with an active connection
 
-The bridge status only constists of 2 different states:
+The bridge status only consists of 2 different states:
 * **online:** Connection to the main compute unit
 * **offline:** No connection to the main compute unit
 

{openmodule-13.1.4 → openmodule-13.3.0}/docs/csv_export.md

@@ -1,5 +1,7 @@
 # CSV Exporter
 
+[TOC]
+
 The file `utils/csv_export.py` contains a `render` function that can be used to export a list of objects or dicts to a 
 csv file. 
 Important specifications of the generated csv file:
@@ -9,7 +11,7 @@ Important specifications of the generated csv file:
 * delimiter is `"\t"` 
 * line terminator is `"\r\n"`. 
 
-Important note: None values are converted to empty strings.
+** ⚠ Important:** `None` values are converted to empty strings.
 
 ## Usage
 
@@ -151,3 +153,8 @@ def main():
     # we create an export on startup, because you can check the export and the cronjob overwrites it anyway
     tasks.create_export()
 ```
+
+## Uploading a file
+
+For uploading files into the Cloud the Databox service is used.
+You can find the documentation of the databox service [here](utils.md?#databox-upload)

{openmodule-13.1.4 → openmodule-13.3.0}/docs/database.md

@@ -1,7 +1,9 @@
 # Database
 
-On the device we use `sqlalchemy` for our database. Additionally we use `alembic` for our migrations. An example of a
-database is included in openmodule-test
+[TOC]
+
+On the device we use `sqlalchemy` for our database, additionally we use `alembic` for our migrations. An example of a
+database is included in openmodule-test.
 
 ## Models
 
@@ -65,16 +67,21 @@ To generate the migration use the provided script `src/database/makemigration.sh
 
 Afterwards the migrations are created under `src/database/alembic/versions`.
 
-### Wo how to migrate
+### How to migrate
 
 The database is always automatically migrated on creation of any database object in the program.
 
+**ℹ Information:** Before a migration upgrade is executed a backup of the current database is created.
+
+The backup file is stored in the database folder and is named `{service-name}_YYYYmmddHHMMSS.sqlite3.backup`.
+Example: "om_showcase_example_1_20240125123456.sqlite3.backup"
+
 ### Migration Examples
 
 The alembic documentation already has some migration examples and tutorials, but here are some quick tips and pitfalls
 we have encountered.
 
-**Renaming Columns**
+**⚠ Renaming Columns**
 
 When you rename a column, alembic usually creates a new column and drops the old one. It does not know whether you
 actually want a new
@@ -144,39 +151,44 @@ with database as db:
 If an exception is raised during an open database session, all changes will be rolled back.  
 It is also possible to create multiple database within a program, but the need to is quite questionable.
 
-## Basic db stuff
+## Basic database operations
 
 Normally sqlalchemy functions should suffice for most jobs. We implemented some additional functionality as functions
 under `openmodule.utils.db_helper`.
 
-### create
+### Create
 
 * db.add(model: DatabaseModel)
 * db.add_all(models: List[DatabaseModel])
 
-### query
+### Query
 
 * base_query = db.query(DatabaseModel)
 * query = base_query..filter_by(**kwargs)
 * query.all() returns a list
 
-### query single object
+### Query results
 
 * instance = query.one() -> returns element or raises exception if query has more or no elements (MultipleResultsFound,
   NoResultFound)
 * instance = query.first() -> returns first element of query or None
+* query.all() -> return all elements of object
+* query.one() -> returns one element or raises exception if no elements
+* query.one_or_none() -> returns first element or None if no elements, but raises an exception if multiple elements
+* query.get() -> query object with primary key, returns None if not found
+* query.count() -> returns number of elements in query
 
-### update
+### Update
 
 * db.add(model: DatabaseModel) -> previously created model
 * db_helper.update_query(db, query, values: dict)
 
-### delete
+### Delete
 
 * db.delete(model: DatabaseModel)
 * db_helper.delete_query(db, query)
 
-### Useful query stuff
+### Useful query functions
 
 * order_by(*args) -> ordering
 * yield_by(int) -> for batch processing

openmodule-13.3.0/docs/deprecated.md

@@ -0,0 +1,56 @@
+# Deprecated
+
+[TOC]
+
+This file holds documentation of deprecated features. It is kept for reference and maybe will be removed in the future.
+
+## Deprecated Utils
+
+### Api (**DEPRECATED**)
+
+We implemented a very basic Api class you can use for http request and that handles errors and authentication. Either
+inherit it or create a class.
+
+```python
+api = Api(**kwargs)
+try:
+    res = api.post("some_url", payload=stuff)
+except ApiException as e:
+    if e.retry:  # <- makes sense to try again - timeouts or server not available ...
+        ...
+```
+
+### Backend (**DEPRECATED**)
+
+There is also a basic implementation of a backend that provides registration and message passing.
+
+```python
+class MyAccessService(AccessService):
+    def __init__(self):
+        super().__init__(implements_session_handling=...)
+        ...
+
+    def rpc_check_access(self, request: AccessRequest) -> AccessCheckResponse:
+        ...
+
+    # session handling
+    def check_in_session(self, message: SessionStartMessage):
+        ...
+
+    def check_out_session(self, message: SessionFinishMessage):
+        ...
+
+    def session_error_message(self, message: Union[SessionDeleteMessage, SessionIncompleteMessage,
+                                                   SessionExitWithoutEntryMessage]):
+        ...
+```
+
+### Alerting (**DEPRECATED**)
+
+The new core also includes an alert handler.
+
+```python
+core.alerts.send(...)
+alert_id = core.alerts.get_or_add_alert_id(...)
+core.alerts.send_with_alert_id(alert_id, ...)
+```