dsa_helpers 3.2.1__tar.gz

dsa_helpers-3.2.1/.gitignore

@@ -0,0 +1,18 @@
+# Python environment directories.
+env/
+pyenv/
+.venv/
+
+*.pyc
+*.egg-info
+dist/
+.env
+
+# Ignore data directory created for testing the package.
+tests/data/
+tests/models/
+
+temp*
+
+large_image-*.whl
+*.pt

dsa_helpers-3.2.1/.readthedocs.yaml

@@ -0,0 +1,16 @@
+version: 2
+
+build:
+  os: "ubuntu-24.04"
+  tools:
+    python: "3.12"
+  jobs:
+    pre_build:
+      - sphinx-apidoc -o docs/source dsa_helpers/
+
+python:
+  install:
+    - requirements: docs/requirements.txt
+
+sphinx:
+  configuration: docs/source/conf.py

dsa_helpers-3.2.1/LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2018 The Python Packaging Authority
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

dsa_helpers-3.2.1/PKG-INFO

@@ -0,0 +1,69 @@
+Metadata-Version: 2.4
+Name: dsa_helpers
+Version: 3.2.1
+Summary: Utility functions for working with the DSA girder client.
+Author-email: "Juan C. Vizcarra" <jvizcar@emory.edu>
+Maintainer-email: "Juan C. Vizcarra" <jvizcar@emory.edu>
+Project-URL: Repository, https://github.com/Gutman-Lab/dsa-python-utils
+Keywords: dsa,girder,girder_client
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: matplotlib
+Requires-Dist: girder-client
+Requires-Dist: numpy
+Requires-Dist: setuptools
+Requires-Dist: pandas
+Requires-Dist: pymongo
+Requires-Dist: pillow
+Requires-Dist: dash
+Requires-Dist: dash-bootstrap-components
+Requires-Dist: shapely
+Requires-Dist: geopandas
+Requires-Dist: rasterio
+Requires-Dist: colorama
+Requires-Dist: histomicstk
+Requires-Dist: pykdtree
+Provides-Extra: opencv
+Requires-Dist: opencv-python; extra == "opencv"
+Provides-Extra: opencv-headless
+Requires-Dist: opencv-python-headless; extra == "opencv-headless"
+Provides-Extra: ultralytics
+Requires-Dist: ultralytics; extra == "ultralytics"
+Provides-Extra: segformer
+Requires-Dist: transformers[torch]; extra == "segformer"
+Requires-Dist: datasets; extra == "segformer"
+Requires-Dist: torchvision; extra == "segformer"
+Requires-Dist: albumentations; extra == "segformer"
+Provides-Extra: all
+Requires-Dist: opencv-python-headless; extra == "all"
+Requires-Dist: ultralytics; extra == "all"
+Requires-Dist: transformers[torch]; extra == "all"
+Requires-Dist: datasets; extra == "all"
+Requires-Dist: torchvision; extra == "all"
+Requires-Dist: albumentations; extra == "all"
+Dynamic: license-file
+
+# Digital Slide Archive (DSA) Helpers
+Digital Slide Archive Python utility library.
+
+This library is available for installation through [Python Package Index (PyPI)](https://pypi.org/).
+
+This library was tested using Python version 3.11.8 and uses the dependencies described in requirements.txt.
+
+This Python PyPI package is found [here](https://pypi.org/project/dsa-helpers/).
+
+## Installation instructions
+DSA Helpers depends on large image and large image eager iterator and OpenCV, which you must install separately. Also, you must install the eager iterator from wheel and it should be installed at the end. Python 3.12 or below is supported, Python 3.14 is not currently supported!
+
+1. Install large with all tile sources:
+```
+$ pip install large-image[all] --find-links https://girder.github.io/large_image_wheels
+```
+2. Install OpenCV, either the headless version (better for running in Docker) or full version
+3. Install dsa-helpers from pip
+4. Pip install the large image with eager iterator wheel, use the --force flag and you can safely igonore the warnings that you get about some large image library compatibilities
+
+A readthedocs for this library can be found [here](https://david-andrew-gutman-dsa-helpers.readthedocs-hosted.com/en/latest/index.html).

dsa_helpers-3.2.1/README.md

@@ -0,0 +1,21 @@
+# Digital Slide Archive (DSA) Helpers
+Digital Slide Archive Python utility library.
+
+This library is available for installation through [Python Package Index (PyPI)](https://pypi.org/).
+
+This library was tested using Python version 3.11.8 and uses the dependencies described in requirements.txt.
+
+This Python PyPI package is found [here](https://pypi.org/project/dsa-helpers/).
+
+## Installation instructions
+DSA Helpers depends on large image and large image eager iterator and OpenCV, which you must install separately. Also, you must install the eager iterator from wheel and it should be installed at the end. Python 3.12 or below is supported, Python 3.14 is not currently supported!
+
+1. Install large with all tile sources:
+```
+$ pip install large-image[all] --find-links https://girder.github.io/large_image_wheels
+```
+2. Install OpenCV, either the headless version (better for running in Docker) or full version
+3. Install dsa-helpers from pip
+4. Pip install the large image with eager iterator wheel, use the --force flag and you can safely igonore the warnings that you get about some large image library compatibilities
+
+A readthedocs for this library can be found [here](https://david-andrew-gutman-dsa-helpers.readthedocs-hosted.com/en/latest/index.html).

dsa_helpers-3.2.1/docker/2.0.6-pytorch2.9.0-cuda13.0-cudnn9/Dockerfile

@@ -0,0 +1,14 @@
+FROM pytorch/pytorch:2.9.0-cuda13.0-cudnn9-runtime
+
+ENV PYTHONUNBUFFERED=TRUE
+ARG DEBIAN_FRONTEND=noninteractive
+
+# Install large image and its tile sources.
+RUN pip install large-image[all] --find-links https://girder.github.io/large_image_wheels
+
+# Install dsa-helpers.
+RUN pip install --no-cache-dir dsa-helpers==2.0.6
+
+# Create a cache directory.
+RUN mkdir /.cache
+RUN chmod 777 /.cache

dsa_helpers-3.2.1/docker/README.md

@@ -0,0 +1,2 @@
+# Docker
+dsa-helpers image hosted on jvizcar account. Subfolders found for each version of the image.

dsa_helpers-3.2.1/docs/requirements.txt

@@ -0,0 +1,16 @@
+girder-client
+numpy
+setuptools
+opencv-python
+pandas
+pymongo
+pillow
+dash
+dash-bootstrap-components
+shapely
+geopandas
+transformers[torch]
+datasets
+torchvision
+large-image
+sphinx_rtd_theme

dsa_helpers-3.2.1/docs/source/conf.py

@@ -0,0 +1,42 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# For the full list of built-in configuration values, see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+import os
+import sys
+
+sys.path.insert(0, os.path.abspath("../.."))
+
+# -- Project information -----------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
+
+project = "DSA Helpers"
+copyright = "2024, Juan Carlos Vizcarra"
+author = "Juan Carlos Vizcarra"
+release = "00.00.01"
+
+# -- General configuration ---------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
+
+extensions = [
+    "sphinx.ext.autodoc",
+    "sphinx.ext.viewcode",
+    "sphinx.ext.napoleon",
+]
+
+templates_path = ["_templates"]
+exclude_patterns = []
+
+
+# -- Options for HTML output -------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
+
+html_theme = "sphinx_rtd_theme"
+html_static_path = ["_static"]

dsa_helpers-3.2.1/docs/source/index.rst

@@ -0,0 +1,18 @@
+.. DSA Helpers documentation master file, created by
+   sphinx-quickstart on Fri Dec  6 12:07:31 2024.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+DSA Helpers documentation
+=========================
+
+Add your content using ``reStructuredText`` syntax. See the
+`reStructuredText <https://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html>`_
+documentation for details.
+
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents:
+
+   modules

dsa_helpers-3.2.1/dsa_helpers/__init__.py

@@ -0,0 +1,9 @@
+# Shadow imports.
+from .imread import imread
+from .imwrite import imwrite
+
+# Version of the dsa-helpers package
+__version__ = "3.2.1"
+
+# To avoid slow downs, do not allow from dsa_helpers import * to import anything.
+__all__ = []

dsa_helpers-3.2.1/dsa_helpers/cli_utils.py

@@ -0,0 +1,183 @@
+from collections import deque
+import psutil, threading, time
+import numpy as np
+
+
+class SystemMonitor:
+    def __init__(self, interval=0.5):
+        self.interval = interval
+        self.cpu_percentages = deque(maxlen=100)
+        self.memory_usage = deque(maxlen=100)
+        self.io_wait = deque(maxlen=100)
+        self.cpu_per_core = deque(maxlen=100)  # Track per-core usage
+        self.memory_bandwidth = deque(maxlen=100)
+        self.io_stats = deque(maxlen=100)
+        self.stop_flag = False
+        self.process = psutil.Process()
+
+    def start(self):
+        self.stop_flag = False
+        self.monitor_thread = threading.Thread(target=self._monitor)
+        self.monitor_thread.start()
+
+    def stop(self):
+        self.stop_flag = True
+        self.monitor_thread.join()
+
+        # Calculate statistics with safety checks
+        cpu_stats = {
+            "mean": (
+                np.mean(self.cpu_percentages)
+                if len(self.cpu_percentages) > 0
+                else 0
+            ),
+            "max": (
+                np.max(self.cpu_percentages)
+                if len(self.cpu_percentages) > 0
+                else 0
+            ),
+            "min": (
+                np.min(self.cpu_percentages)
+                if len(self.cpu_percentages) > 0
+                else 0
+            ),
+            "per_core_mean": (
+                np.mean(self.cpu_per_core, axis=0)
+                if len(self.cpu_per_core) > 0
+                else np.zeros(psutil.cpu_count())
+            ),
+            "per_core_max": (
+                np.max(self.cpu_per_core, axis=0)
+                if len(self.cpu_per_core) > 0
+                else np.zeros(psutil.cpu_count())
+            ),
+            "active_cores_mean": (
+                np.mean(
+                    [
+                        sum(1 for x in cores if x > 10)
+                        for cores in self.cpu_per_core
+                    ]
+                )
+                if len(self.cpu_per_core) > 0
+                else 0
+            ),
+        }
+
+        mem_stats = {
+            "mean": (
+                np.mean(self.memory_usage) if len(self.memory_usage) > 0 else 0
+            ),
+            "max": (
+                np.max(self.memory_usage) if len(self.memory_usage) > 0 else 0
+            ),
+            "min": (
+                np.min(self.memory_usage) if len(self.memory_usage) > 0 else 0
+            ),
+        }
+
+        io_stats = {
+            "wait_mean": np.mean(self.io_wait) if len(self.io_wait) > 0 else 0,
+            "wait_max": np.max(self.io_wait) if len(self.io_wait) > 0 else 0,
+            "throughput_read": (
+                np.mean(
+                    [
+                        sum(disk["read_bytes"] for disk in x["disk"].values())
+                        / x["time_delta"]
+                        for x in self.io_stats
+                    ]
+                )
+                / 1024
+                / 1024
+                if len(self.io_stats) > 0
+                else 0
+            ),
+            "throughput_write": (
+                np.mean(
+                    [
+                        sum(disk["write_bytes"] for disk in x["disk"].values())
+                        / x["time_delta"]
+                        for x in self.io_stats
+                    ]
+                )
+                / 1024
+                / 1024
+                if len(self.io_stats) > 0
+                else 0
+            ),
+            "network_read": (
+                np.mean(
+                    [
+                        x["network"]["bytes_recv"] / x["time_delta"]
+                        for x in self.io_stats
+                    ]
+                )
+                / 1024
+                / 1024
+                if len(self.io_stats) > 0
+                else 0
+            ),
+            "network_write": (
+                np.mean(
+                    [
+                        x["network"]["bytes_sent"] / x["time_delta"]
+                        for x in self.io_stats
+                    ]
+                )
+                / 1024
+                / 1024
+                if len(self.io_stats) > 0
+                else 0
+            ),
+        }
+
+        return {"cpu": cpu_stats, "memory": mem_stats, "io": io_stats}
+
+    def _monitor(self):
+        last_io = psutil.disk_io_counters(perdisk=True)  # Monitor per-disk
+        last_net = psutil.net_io_counters()  # Add network monitoring
+        last_time = time.time()
+
+        while not self.stop_flag:
+            # CPU usage (overall and per-core)
+            per_core = psutil.cpu_percent(percpu=True)
+            self.cpu_percentages.append(np.mean(per_core))
+            self.cpu_per_core.append(per_core)
+
+            # Memory usage
+            mem = self.process.memory_info()
+            self.memory_usage.append(mem.rss / 1024 / 1024)  # Convert to MB
+
+            # IO wait and throughput
+            cpu_times = psutil.cpu_times_percent()
+            self.io_wait.append(cpu_times.iowait)
+
+            # Enhanced IO monitoring
+            current_io = psutil.disk_io_counters(perdisk=True)
+            current_net = psutil.net_io_counters()
+            current_time = time.time()
+
+            # Calculate both disk and network throughput
+            delta_time = current_time - last_time
+            io_delta = {
+                "disk": {
+                    disk: {
+                        "read_bytes": current_io[disk].read_bytes
+                        - last_io[disk].read_bytes,
+                        "write_bytes": current_io[disk].write_bytes
+                        - last_io[disk].write_bytes,
+                    }
+                    for disk in current_io
+                },
+                "network": {
+                    "bytes_recv": current_net.bytes_recv - last_net.bytes_recv,
+                    "bytes_sent": current_net.bytes_sent - last_net.bytes_sent,
+                },
+                "time_delta": delta_time,
+            }
+            self.io_stats.append(io_delta)
+
+            last_io = current_io
+            last_net = current_net
+            last_time = current_time
+
+            time.sleep(self.interval)

dsa_helpers-3.2.1/dsa_helpers/dash/__init__.py

@@ -0,0 +1,3 @@
+from . import header
+
+__all__ = []

dsa_helpers-3.2.1/dsa_helpers/dash/dash_paperdragon_utils.py

@@ -0,0 +1,76 @@
+def format_ann_docs_for_paperjs(
+    geojson_ann_docs: list[dict], return_as_input_to_paper: bool = False
+) -> list[dict]:
+    """Format DSA annotations in the geojson format into a format that paperdragon
+    can read and show as polygons.
+
+    Args:
+        geojson_ann_docs (list[dict]): A list of geojson annotation documents.
+        return_as_input_to_paper (bool): Whether to return the formatted documents as
+            input to paper. Defaults to False.
+
+    Returns:
+        list[dict]: A list of formatted annotation documents for paperjs. Note that
+            it will return an empty list if the documents could not be formatted.
+
+    """
+    # NOTE: hard-coded to only do the first doc.
+    formatted_docs = []
+
+    # Loop through each annotation document.
+    for ann_doc in geojson_ann_docs:
+        # Grab the features of this document, contains the polygon info.
+        features_to_include = []
+
+        for feature in ann_doc["features"]:
+            geometry = feature["geometry"]
+            if geometry["type"] == "Polygon":
+                properties = feature["properties"]
+                properties["rescale"] = {"strokeWidth": properties["lineWidth"]}
+                properties["strokeColor"] = properties["lineColor"]
+                properties["source"] = "dsa"
+
+                # Paperjs only supports multipolygons.
+                geometry["type"] = "MultiPolygon"
+
+                coordinates = geometry["coordinates"]
+
+                adjusted_coordinates = [
+                    [[[int(coord) for coord in point[:2]] for point in coordinates[0]]]
+                ]
+
+                geometry["coordinates"] = adjusted_coordinates
+
+                features_to_include.append(feature)
+
+        if len(features_to_include):
+            formatted_docs.append(features_to_include)
+
+    if return_as_input_to_paper:
+        return get_input_to_paper_dict(formatted_docs)
+
+    return formatted_docs
+
+
+def get_input_to_paper_dict(formatted_docs: list[dict] | None = None) -> dict:
+    """Get the input to paper dictionary.
+
+    Args:
+        formatted_docs (list[dict]): A list of formatted annotation documents for paperjs.
+
+    Returns:
+        dict: The input to paper dictionary.
+
+    """
+    input_to_paper = [{"type": "clearItems"}]
+
+    if formatted_docs is not None and len(formatted_docs):
+        for doc in formatted_docs:
+            input_to_paper.append(
+                {
+                    "type": "drawItems",
+                    "itemList": doc,
+                }
+            )
+
+    return {"actions": input_to_paper}

dsa_helpers-3.2.1/dsa_helpers/dash/header.py

@@ -0,0 +1,155 @@
+from dash import html, dcc
+import dash_bootstrap_components as dbc
+from .header_callbacks import get_callbacks
+
+
+def get_header(
+    dsa_api_url: str,
+    title: str = "App",
+    store_id: str = "user-store",
+    config: dict | None = None,
+) -> html.Div:
+    """Get the Dash html component that contains the header of the application
+    with login capabilities.
+
+    Args:
+        dsa_api_url (str): The URL of the DSA API.
+        title (str, optional): The title of the application. Defaults to "App".
+        config (dict, optional): The configuration of the component.
+            Defaults to None which uses default settings.
+
+    Returns:
+        html.Div: The header component.
+
+    """
+    if config is None:
+        config = {}
+
+    get_callbacks(dsa_api_url, store_id)
+
+    login_modal = dbc.Modal(
+        [
+            dbc.ModalHeader("Log in"),
+            dbc.ModalBody(
+                [
+                    html.Div(
+                        "Login or email",
+                        style={"margin": 5, "fontWeight": "bold"},
+                    ),
+                    dbc.Input(
+                        id=f"{store_id}=login",
+                        type="text",
+                        placeholder="Enter login",
+                        style={"margin": 5},
+                    ),
+                    html.Div(
+                        "Password",
+                        style={
+                            "margin": 5,
+                            "marginTop": 15,
+                            "fontWeight": "bold",
+                        },
+                    ),
+                    dbc.Input(
+                        id=f"{store_id}=password",
+                        type="password",
+                        placeholder="Enter password",
+                        style={"margin": 5},
+                    ),
+                    html.Div(
+                        "Login failed.",
+                        hidden=True,
+                        id=f"{store_id}=login-failed",
+                        style={
+                            "color": "red",
+                            "fontWeight": "bold",
+                            "margin": 10,
+                        },
+                    ),
+                ],
+            ),
+            dbc.ModalFooter(
+                dbc.Row(
+                    [
+                        dbc.Col(
+                            dbc.Button(
+                                "Close",
+                                id=f"{store_id}=close-login-modal",
+                                className="me-1",
+                                color="light",
+                            )
+                        ),
+                        dbc.Col(
+                            dbc.Button(
+                                "Login",
+                                id=f"{store_id}=log-in-btn",
+                                className="me-1",
+                                color="primary",
+                            )
+                        ),
+                    ],
+                )
+            ),
+        ],
+        is_open=False,
+        id=f"{store_id}=login-modal",
+    )
+
+    logout_modal = dbc.Modal(
+        [
+            dbc.ModalFooter(
+                [
+                    dbc.Button(
+                        "Log out",
+                        id=f"{store_id}=logout-btn",
+                        color="danger",
+                        className="me-1",
+                    ),
+                ]
+            )
+        ],
+        is_open=False,
+        id=f"{store_id}=logout-modal",
+    )
+
+    return html.Div(
+        [
+            dbc.Row(
+                [
+                    dbc.Col(
+                        html.H1(
+                            title,
+                            style={
+                                "fontWeight": "bold",
+                                "color": config.get("titleColor", "#d9d9d6"),
+                                "marginLeft": 5,
+                            },
+                        ),
+                        width="auto",
+                    ),
+                    dbc.Col(
+                        dbc.Button(
+                            "Log in",
+                            id=f"{store_id}=login-btn",
+                            color=config.get("backgroundColor", "#012169"),
+                            style={
+                                "color": config.get("fontColor", "#f2a900"),
+                                "fontSize": "1.2rem",
+                            },
+                            className="me-1",
+                        ),
+                        width="auto",
+                    ),
+                ],
+                justify="between",
+                align="center",
+            ),
+            login_modal,
+            logout_modal,
+            dcc.Store(id=store_id, storage_type="local", data={}),
+        ],
+        style={
+            "backgroundColor": config.get("backgroundColor", "#012169"),
+            "padding": 5,
+        },
+    )