arkindex-base-worker 0.5.2a1__tar.gz

arkindex_base_worker-0.5.2a1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 Teklia
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

arkindex_base_worker-0.5.2a1/PKG-INFO

@@ -0,0 +1,49 @@
+Metadata-Version: 2.4
+Name: arkindex-base-worker
+Version: 0.5.2a1
+Summary: Base Worker to easily build Arkindex ML workflows
+Author-email: Teklia <contact@teklia.com>
+Maintainer-email: Teklia <contact@teklia.com>
+Project-URL: Homepage, https://workers.arkindex.org
+Project-URL: Documentation, https://workers.arkindex.org
+Project-URL: Repository, https://gitlab.teklia.com/workers/base-worker
+Project-URL: Bug Tracker, https://gitlab.teklia.com/workers/base-worker/issues
+Project-URL: Authors, https://teklia.com
+Keywords: python
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: humanize==4.15.0
+Requires-Dist: peewee~=3.17
+Requires-Dist: Pillow==11.3.0
+Requires-Dist: python-gnupg==0.5.6
+Requires-Dist: shapely==2.0.6
+Requires-Dist: teklia-toolbox==0.1.12
+Requires-Dist: zstandard==0.25.0
+Provides-Extra: tests
+Requires-Dist: pytest-mock==3.15.1; extra == "tests"
+Requires-Dist: pytest-responses==0.5.1; extra == "tests"
+Dynamic: license-file
+
+# Arkindex base Worker
+
+An easy to use Python 3 high level API client, to build ML tasks.
+
+This is an open-source project, licensed using [the MIT license](https://opensource.org/license/mit/).
+
+## Documentation
+
+The [documentation](https://workers.arkindex.org/) is made with [Antora](https://antora.org/) and is hosted by [GitLab Pages](https://docs.gitlab.com/ee/user/project/pages/).
+
+## Create a new worker using our template
+
+```shell
+pip install --user cookiecutter
+cookiecutter git@gitlab.teklia.com:workers/base-worker.git
+```

arkindex_base_worker-0.5.2a1/README.md

@@ -0,0 +1,16 @@
+# Arkindex base Worker
+
+An easy to use Python 3 high level API client, to build ML tasks.
+
+This is an open-source project, licensed using [the MIT license](https://opensource.org/license/mit/).
+
+## Documentation
+
+The [documentation](https://workers.arkindex.org/) is made with [Antora](https://antora.org/) and is hosted by [GitLab Pages](https://docs.gitlab.com/ee/user/project/pages/).
+
+## Create a new worker using our template
+
+```shell
+pip install --user cookiecutter
+cookiecutter git@gitlab.teklia.com:workers/base-worker.git
+```

arkindex_base_worker-0.5.2a1/arkindex_base_worker.egg-info/PKG-INFO

@@ -0,0 +1,49 @@
+Metadata-Version: 2.4
+Name: arkindex-base-worker
+Version: 0.5.2a1
+Summary: Base Worker to easily build Arkindex ML workflows
+Author-email: Teklia <contact@teklia.com>
+Maintainer-email: Teklia <contact@teklia.com>
+Project-URL: Homepage, https://workers.arkindex.org
+Project-URL: Documentation, https://workers.arkindex.org
+Project-URL: Repository, https://gitlab.teklia.com/workers/base-worker
+Project-URL: Bug Tracker, https://gitlab.teklia.com/workers/base-worker/issues
+Project-URL: Authors, https://teklia.com
+Keywords: python
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: humanize==4.15.0
+Requires-Dist: peewee~=3.17
+Requires-Dist: Pillow==11.3.0
+Requires-Dist: python-gnupg==0.5.6
+Requires-Dist: shapely==2.0.6
+Requires-Dist: teklia-toolbox==0.1.12
+Requires-Dist: zstandard==0.25.0
+Provides-Extra: tests
+Requires-Dist: pytest-mock==3.15.1; extra == "tests"
+Requires-Dist: pytest-responses==0.5.1; extra == "tests"
+Dynamic: license-file
+
+# Arkindex base Worker
+
+An easy to use Python 3 high level API client, to build ML tasks.
+
+This is an open-source project, licensed using [the MIT license](https://opensource.org/license/mit/).
+
+## Documentation
+
+The [documentation](https://workers.arkindex.org/) is made with [Antora](https://antora.org/) and is hosted by [GitLab Pages](https://docs.gitlab.com/ee/user/project/pages/).
+
+## Create a new worker using our template
+
+```shell
+pip install --user cookiecutter
+cookiecutter git@gitlab.teklia.com:workers/base-worker.git
+```

arkindex_base_worker-0.5.2a1/arkindex_base_worker.egg-info/SOURCES.txt

@@ -0,0 +1,64 @@
+LICENSE
+README.md
+pyproject.toml
+arkindex_base_worker.egg-info/PKG-INFO
+arkindex_base_worker.egg-info/SOURCES.txt
+arkindex_base_worker.egg-info/dependency_links.txt
+arkindex_base_worker.egg-info/requires.txt
+arkindex_base_worker.egg-info/top_level.txt
+arkindex_worker/__init__.py
+arkindex_worker/cache.py
+arkindex_worker/image.py
+arkindex_worker/models.py
+arkindex_worker/utils.py
+arkindex_worker/worker/__init__.py
+arkindex_worker/worker/base.py
+arkindex_worker/worker/classification.py
+arkindex_worker/worker/corpus.py
+arkindex_worker/worker/dataset.py
+arkindex_worker/worker/element.py
+arkindex_worker/worker/entity.py
+arkindex_worker/worker/image.py
+arkindex_worker/worker/metadata.py
+arkindex_worker/worker/process.py
+arkindex_worker/worker/task.py
+arkindex_worker/worker/training.py
+arkindex_worker/worker/transcription.py
+examples/standalone/python/worker.py
+examples/tooled/python/worker.py
+hooks/pre_gen_project.py
+tests/__init__.py
+tests/conftest.py
+tests/test_base_worker.py
+tests/test_cache.py
+tests/test_dataset_worker.py
+tests/test_element.py
+tests/test_image.py
+tests/test_merge.py
+tests/test_modern_config.py
+tests/test_utils.py
+tests/test_elements_worker/__init__.py
+tests/test_elements_worker/test_classification.py
+tests/test_elements_worker/test_cli.py
+tests/test_elements_worker/test_corpus.py
+tests/test_elements_worker/test_dataset.py
+tests/test_elements_worker/test_element.py
+tests/test_elements_worker/test_element_create_multiple.py
+tests/test_elements_worker/test_element_create_single.py
+tests/test_elements_worker/test_element_list_children.py
+tests/test_elements_worker/test_element_list_parents.py
+tests/test_elements_worker/test_entity.py
+tests/test_elements_worker/test_image.py
+tests/test_elements_worker/test_metadata.py
+tests/test_elements_worker/test_process.py
+tests/test_elements_worker/test_task.py
+tests/test_elements_worker/test_training.py
+tests/test_elements_worker/test_transcription_create.py
+tests/test_elements_worker/test_transcription_create_with_elements.py
+tests/test_elements_worker/test_transcription_list.py
+tests/test_elements_worker/test_worker.py
+worker-demo/tests/__init__.py
+worker-demo/tests/conftest.py
+worker-demo/tests/test_worker.py
+worker-demo/worker_demo/__init__.py
+worker-demo/worker_demo/worker.py

arkindex_base_worker-0.5.2a1/arkindex_base_worker.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

arkindex_base_worker-0.5.2a1/arkindex_base_worker.egg-info/requires.txt

@@ -0,0 +1,11 @@
+humanize==4.15.0
+peewee~=3.17
+Pillow==11.3.0
+python-gnupg==0.5.6
+shapely==2.0.6
+teklia-toolbox==0.1.12
+zstandard==0.25.0
+
+[tests]
+pytest-mock==3.15.1
+pytest-responses==0.5.1

arkindex_base_worker-0.5.2a1/arkindex_base_worker.egg-info/top_level.txt

@@ -0,0 +1,7 @@
+arkindex_worker
+dist
+docs
+examples
+hooks
+tests
+worker-demo

arkindex_base_worker-0.5.2a1/arkindex_worker/__init__.py

@@ -0,0 +1,10 @@
+import importlib.metadata
+import logging
+
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s %(levelname)s/%(name)s: %(message)s",
+)
+logger = logging.getLogger(__name__)
+
+VERSION = importlib.metadata.version("arkindex-base-worker")

arkindex_base_worker-0.5.2a1/arkindex_worker/cache.py

@@ -0,0 +1,376 @@
+"""
+Database mappings and helper methods for the experimental worker caching feature.
+
+On methods that support caching, the database will be used for all reads,
+and writes will go both to the Arkindex API and the database,
+reducing network usage.
+"""
+
+import json
+import sqlite3
+from pathlib import Path
+
+from peewee import (
+    SQL,
+    BooleanField,
+    CharField,
+    Check,
+    CompositeKey,
+    Field,
+    FloatField,
+    ForeignKeyField,
+    IntegerField,
+    Model,
+    OperationalError,
+    SqliteDatabase,
+    TextField,
+    UUIDField,
+)
+from PIL import Image
+
+from arkindex_worker import logger
+
+db = SqliteDatabase(None)
+
+
+class JSONField(Field):
+    """
+    A Peewee field that stores a JSON payload as a string and parses it automatically.
+    """
+
+    field_type = "text"
+
+    def db_value(self, value):
+        if value is None:
+            return
+        return json.dumps(value)
+
+    def python_value(self, value):
+        if value is None:
+            return
+        return json.loads(value)
+
+
+class Version(Model):
+    """
+    Cache version table, used to warn about incompatible cache databases
+    when a worker uses an outdated version of ``base-worker``.
+    """
+
+    version = IntegerField(primary_key=True)
+
+    class Meta:
+        database = db
+        table_name = "version"
+
+
+class CachedImage(Model):
+    """
+    Cache image table
+    """
+
+    id = UUIDField(primary_key=True)
+    width = IntegerField()
+    height = IntegerField()
+    url = TextField()
+    version = IntegerField(default=2)
+
+    class Meta:
+        database = db
+        table_name = "images"
+
+
+class CachedElement(Model):
+    """
+    Cache element table
+    """
+
+    id = UUIDField(primary_key=True)
+    parent_id = UUIDField(null=True)
+    type = CharField(max_length=50)
+    image = ForeignKeyField(CachedImage, backref="elements", null=True)
+    polygon = JSONField(null=True)
+    rotation_angle = IntegerField(default=0)
+    mirrored = BooleanField(default=False)
+    initial = BooleanField(default=False)
+    # Needed to filter elements with cache
+    worker_version_id = UUIDField(null=True)
+    worker_run_id = UUIDField(null=True)
+    confidence = FloatField(null=True)
+
+    class Meta:
+        database = db
+        table_name = "elements"
+
+    def open_image(
+        self,
+        *args,
+        max_width: int | None = None,
+        max_height: int | None = None,
+        **kwargs,
+    ) -> Image:
+        """
+        Open this element's image as a Pillow image.
+        This does not crop the image to the element's polygon.
+        IIIF servers with maxWidth, maxHeight or maxArea restrictions on image size are not supported.
+
+        Warns:
+        ----
+           If both, ``max_width`` and ``max_height`` are set, the image ratio is not preserved.
+
+
+        :param *args: Positional arguments passed to [arkindex_worker.image.open_image][]
+        :param max_width: The maximum width of the image.
+        :param max_height: The maximum height of the image.
+        :param **kwargs: Keyword arguments passed to [arkindex_worker.image.open_image][]
+        :raises ValueError: When this element does not have an image ID or a polygon.
+        :return: A Pillow image.
+        """
+        from arkindex_worker.image import open_image, polygon_bounding_box
+
+        if not self.image_id or not self.polygon:
+            raise ValueError(f"Element {self.id} has no image")
+
+        # Always fetch the image from the bounding box when size differs from full image
+        bounding_box = polygon_bounding_box(self.polygon)
+        if (
+            bounding_box.width != self.image.width
+            or bounding_box.height != self.image.height
+        ):
+            box = f"{bounding_box.x},{bounding_box.y},{bounding_box.width},{bounding_box.height}"
+        else:
+            box = "full"
+
+        if max_width is None and max_height is None:
+            resize = "full"
+        else:
+            if (
+                # Do not resize for polygons that do not exactly match the images
+                # as the resize is made directly by the IIIF server using the box parameter
+                bounding_box.width != self.image.width
+                or bounding_box.height != self.image.height
+            ) or (
+                # Do not resize when the image is below the maximum size
+                (max_width is None or self.image.width <= max_width)
+                and (max_height is None or self.image.height <= max_height)
+            ):
+                resize = "full"
+            else:
+                resize = f"{max_width or ''},{max_height or ''}"
+
+        # Use `max` instead of `full` for IIIF 3, since `full` was deprecated in 2.1 then removed in 3.0
+        if self.image.version == 3 and resize == "full":
+            resize = "max"
+
+        url = self.image.url
+        if not url.endswith("/"):
+            url += "/"
+
+        return open_image(
+            f"{url}{box}/{resize}/0/default.jpg",
+            *args,
+            rotation_angle=self.rotation_angle,
+            mirrored=self.mirrored,
+            **kwargs,
+        )
+
+
+class CachedTranscription(Model):
+    """
+    Cache transcription table
+    """
+
+    id = UUIDField(primary_key=True)
+    element = ForeignKeyField(CachedElement, backref="transcriptions")
+    text = TextField()
+    confidence = FloatField(null=True)
+    orientation = CharField(max_length=50)
+    # Needed to filter transcriptions with cache
+    worker_version_id = UUIDField(null=True)
+    worker_run_id = UUIDField(null=True)
+
+    class Meta:
+        database = db
+        table_name = "transcriptions"
+
+
+class CachedClassification(Model):
+    """
+    Cache classification table
+    """
+
+    id = UUIDField(primary_key=True)
+    element = ForeignKeyField(CachedElement, backref="classifications")
+    class_name = TextField()
+    confidence = FloatField()
+    state = CharField(max_length=10)
+    worker_run_id = UUIDField(null=True)
+
+    class Meta:
+        database = db
+        table_name = "classifications"
+
+
+class CachedTranscriptionEntity(Model):
+    """
+    Cache transcription entity table
+    """
+
+    transcription = ForeignKeyField(
+        CachedTranscription, backref="transcription_entities"
+    )
+    type = CharField(max_length=50)
+    offset = IntegerField(constraints=[Check("offset >= 0")])
+    length = IntegerField(constraints=[Check("length > 0")])
+    worker_run_id = UUIDField(null=True)
+    confidence = FloatField(null=True)
+
+    class Meta:
+        primary_key = CompositeKey("transcription", "type")
+        database = db
+        table_name = "transcription_entities"
+
+
+class CachedDataset(Model):
+    id = UUIDField(primary_key=True)
+    name = CharField()
+    state = CharField(constraints=[SQL("DEFAULT 'open'")])
+    sets = TextField()
+
+    class Meta:
+        database = db
+        table_name = "datasets"
+
+
+class CachedDatasetElement(Model):
+    id = UUIDField(primary_key=True)
+    element = ForeignKeyField(column_name="element_id", field="id", model=CachedElement)
+    dataset = ForeignKeyField(column_name="dataset_id", field="id", model=CachedDataset)
+    set_name = CharField()
+
+    class Meta:
+        database = db
+        table_name = "dataset_elements"
+
+
+# Add all the managed models in that list
+# It's used here, but also in unit tests
+MODELS = [
+    CachedImage,
+    CachedElement,
+    CachedTranscription,
+    CachedClassification,
+    CachedTranscriptionEntity,
+    CachedDataset,
+    CachedDatasetElement,
+]
+SQL_VERSION = 5
+
+
+def init_cache_db(path: Path):
+    """
+    Create the cache database on the given path
+    :param path: Where the new database should be created
+    """
+    db.init(
+        path,
+        pragmas={
+            # SQLite ignores foreign keys and check constraints by default!
+            "foreign_keys": 1,
+            "ignore_check_constraints": 0,
+        },
+    )
+    db.connect()
+    logger.info(f"Connected to cache on {path}")
+
+
+def create_tables():
+    """
+    Creates the tables in the cache DB only if they do not already exist.
+    """
+    db.create_tables(MODELS)
+
+
+def create_version_table():
+    """
+    Creates the Version table in the cache DB.
+    This step must be independent from other tables creation since we only
+    want to create the table and add the one and only Version entry when the
+    cache is created from scratch.
+    """
+    db.create_tables([Version])
+    Version.create(version=SQL_VERSION)
+
+
+def check_version(cache_path: str | Path):
+    """
+    Check the validity of the SQLite version
+
+    :param cache_path: Path towards a local SQLite database
+    """
+    with SqliteDatabase(cache_path) as provided_db, provided_db.bind_ctx([Version]):
+        try:
+            version = Version.get().version
+        except OperationalError:
+            version = None
+
+        assert version == SQL_VERSION, (
+            f"The SQLite database {cache_path} does not have the correct cache version, it should be {SQL_VERSION}"
+        )
+
+
+def merge_parents_cache(paths: list, current_database: Path):
+    """
+    Merge all the potential parent task's databases into the existing local one
+    :param paths: Path to cache databases
+    :param current_database: Path to the current database
+    """
+    assert current_database.exists()
+
+    if not paths:
+        logger.info("No parents cache to use")
+        return
+
+    # Open a connection on current database
+    connection = sqlite3.connect(current_database)
+    cursor = connection.cursor()
+
+    # Merge each table into the local database
+    for idx, path in enumerate(paths):
+        # Check that the parent cache uses a compatible version
+        check_version(path)
+
+        with SqliteDatabase(path) as source, source.bind_ctx(MODELS):
+            source.create_tables(MODELS)
+
+        logger.info(f"Merging parent db {path} into {current_database}")
+        statements = [
+            "PRAGMA page_size=80000;",
+            "PRAGMA synchronous=OFF;",
+            f"ATTACH DATABASE '{path}' AS source_{idx};",
+            f"REPLACE INTO images SELECT * FROM source_{idx}.images;",
+            f"REPLACE INTO elements SELECT * FROM source_{idx}.elements;",
+            f"REPLACE INTO transcriptions SELECT * FROM source_{idx}.transcriptions;",
+            f"REPLACE INTO classifications SELECT * FROM source_{idx}.classifications;",
+            f"REPLACE INTO transcription_entities SELECT * FROM source_{idx}.transcription_entities;",
+            f"REPLACE INTO datasets SELECT * FROM source_{idx}.datasets;",
+            f"REPLACE INTO dataset_elements SELECT * FROM source_{idx}.dataset_elements;",
+        ]
+
+        for statement in statements:
+            cursor.execute(statement)
+        connection.commit()
+
+
+def unsupported_cache(func):
+    def wrapper(self, *args, **kwargs):
+        results = func(self, *args, **kwargs)
+
+        if self.use_cache:
+            logger.warning(
+                f"This API helper `{func.__name__}` did not update the cache database"
+            )
+
+        return results
+
+    return wrapper