mindtrace 0.1.0__tar.gz

mindtrace-0.1.0/PKG-INFO

@@ -0,0 +1,121 @@
+Metadata-Version: 2.4
+Name: mindtrace
+Version: 0.1.0
+Summary: Mindtrace monorepo with modular packages
+Author: Mindtrace Team
+License-Expression: Apache-2.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: mindtrace-core>=0.1.0
+Requires-Dist: mindtrace-jobs>=0.1.0
+Requires-Dist: mindtrace-registry>=0.1.0
+Requires-Dist: mindtrace-database>=0.1.0
+Requires-Dist: mindtrace-services>=0.1.0
+Requires-Dist: mindtrace-hardware>=0.1.0
+Requires-Dist: mindtrace-cluster>=0.1.0
+Requires-Dist: mindtrace-models>=0.1.0
+Requires-Dist: mindtrace-automation>=0.1.0
+Requires-Dist: mindtrace-apps>=0.1.0
+Requires-Dist: mindtrace-ui>=0.1.0
+Requires-Dist: mindtrace-datalake>=0.1.0
+Requires-Dist: mindtrace-storage>=0.1.0
+
+# Mindtrace Module Dependency Structure
+
+Mindtrace is organized into a layered workspace to support ML components as Python modules with clearly defined boundaries and dependencies.
+
+---
+
+## 📐 Layered Architecture
+
+We use a level-based system for organizing modules based on dependency direction and build order.
+
+### **Level 1: Core**
+- `core`: Foundational utilities and base classes used across all other modules.
+
+### **Level 2: Core Consumers**
+- `jobs`: Job execution and backend interfaces.
+- `registry`: Artifact and metadata management.
+- `database`: Redis, Mongo, and DB access layers.
+- `services`: Service base classes, authentication, and gateways.
+- `ui`: Optional UI libraries and components.
+
+### **Level 3: Infrastructure Modules**
+- `hardware`: Interfaces for cameras, PLCs, scanners, etc.
+- `cluster`: Runtime cluster management, nodes, and workers.
+- `datalake`: Dataset interfaces for HuggingFace and Mindtrace datasets.
+- `models`: Core model definitions and leaderboard utilities.
+
+### **Level 4: Automation**
+- `automation`: Integration of pipelines and orchestration using level 2–3 modules.
+
+### **Level 5: Applications**
+- `apps`: End-user applications composed of all previous levels.
+  - E.g., Demo pipelines
+
+---
+
+## 🔄 Dependency Flow
+
+Each layer only depends on modules in lower levels.
+
+| Module     | Depends On                                           |
+|------------|------------------------------------------------------|
+| `core`     | –                                                    |
+| `jobs`     | `core`, `services`                                   |
+| `registry` | `core`                                               |
+| `database` | `core`                                               |
+| `services` | `core`                                               |
+| `ui`       | `core`                                               |
+| `cluster`  | `jobs`, `registry`, `database`, `services`           |
+| `datalake` | `registry`, `database`, `services`                   |
+| `models`   | `registry`, `services`                               |
+| `automation` | `jobs`, `registry`, `database`, `services`, `datalake`, `models`, `cluster` |
+| `apps`     | Everything                                           |
+
+---
+
+
+## 🛠️ Build
+
+Building wheels and source distributions, from the root of the repo:  
+```bash
+uv build --all-packages
+ls dist/
+```
+For building only wheels:  
+```bash
+uv build --all-packages --wheel
+ls dist/
+```
+They may then be installed in a new venv (the entire `mindtrace` package or any submodule `mindtrace-core`) via:  
+```bash
+uv pip install mindtrace --find-links /path/to/dist
+# or
+uv pip install /path/to/dist/mindtrace.whl
+```
+Note: You may need to use `uv pip install --force-reinstall` in case you encounter `ModuleNotFoundError`.  
+Checking the installation:  
+```bash
+uv run python -c "from mindtrace.core import Mindtrace; print('OK')"
+```
+
+
+## 🛠️ Usage Examples
+
+Installing the full Mindtrace package:
+```bash
+uv add mindtrace
+```
+Installing a minimal dependency chain (e.g., for Datalake development):
+```bash
+uv add mindtrace-datalake
+```
+Python Imports
+```python
+from mindtrace import core, registry, database, services
+```
+
+

mindtrace-0.1.0/README.md

@@ -0,0 +1,97 @@
+# Mindtrace Module Dependency Structure
+
+Mindtrace is organized into a layered workspace to support ML components as Python modules with clearly defined boundaries and dependencies.
+
+---
+
+## 📐 Layered Architecture
+
+We use a level-based system for organizing modules based on dependency direction and build order.
+
+### **Level 1: Core**
+- `core`: Foundational utilities and base classes used across all other modules.
+
+### **Level 2: Core Consumers**
+- `jobs`: Job execution and backend interfaces.
+- `registry`: Artifact and metadata management.
+- `database`: Redis, Mongo, and DB access layers.
+- `services`: Service base classes, authentication, and gateways.
+- `ui`: Optional UI libraries and components.
+
+### **Level 3: Infrastructure Modules**
+- `hardware`: Interfaces for cameras, PLCs, scanners, etc.
+- `cluster`: Runtime cluster management, nodes, and workers.
+- `datalake`: Dataset interfaces for HuggingFace and Mindtrace datasets.
+- `models`: Core model definitions and leaderboard utilities.
+
+### **Level 4: Automation**
+- `automation`: Integration of pipelines and orchestration using level 2–3 modules.
+
+### **Level 5: Applications**
+- `apps`: End-user applications composed of all previous levels.
+  - E.g., Demo pipelines
+
+---
+
+## 🔄 Dependency Flow
+
+Each layer only depends on modules in lower levels.
+
+| Module     | Depends On                                           |
+|------------|------------------------------------------------------|
+| `core`     | –                                                    |
+| `jobs`     | `core`, `services`                                   |
+| `registry` | `core`                                               |
+| `database` | `core`                                               |
+| `services` | `core`                                               |
+| `ui`       | `core`                                               |
+| `cluster`  | `jobs`, `registry`, `database`, `services`           |
+| `datalake` | `registry`, `database`, `services`                   |
+| `models`   | `registry`, `services`                               |
+| `automation` | `jobs`, `registry`, `database`, `services`, `datalake`, `models`, `cluster` |
+| `apps`     | Everything                                           |
+
+---
+
+
+## 🛠️ Build
+
+Building wheels and source distributions, from the root of the repo:  
+```bash
+uv build --all-packages
+ls dist/
+```
+For building only wheels:  
+```bash
+uv build --all-packages --wheel
+ls dist/
+```
+They may then be installed in a new venv (the entire `mindtrace` package or any submodule `mindtrace-core`) via:  
+```bash
+uv pip install mindtrace --find-links /path/to/dist
+# or
+uv pip install /path/to/dist/mindtrace.whl
+```
+Note: You may need to use `uv pip install --force-reinstall` in case you encounter `ModuleNotFoundError`.  
+Checking the installation:  
+```bash
+uv run python -c "from mindtrace.core import Mindtrace; print('OK')"
+```
+
+
+## 🛠️ Usage Examples
+
+Installing the full Mindtrace package:
+```bash
+uv add mindtrace
+```
+Installing a minimal dependency chain (e.g., for Datalake development):
+```bash
+uv add mindtrace-datalake
+```
+Python Imports
+```python
+from mindtrace import core, registry, database, services
+```
+
+

mindtrace-0.1.0/mindtrace/core/mindtrace/core/__init__.py

@@ -0,0 +1,34 @@
+from mindtrace.core.base import Mindtrace, MindtraceABC, MindtraceMeta
+from mindtrace.core.config import Config
+from mindtrace.core.logging.logger import setup_logger
+from mindtrace.core.observables.context_listener import ContextListener
+from mindtrace.core.observables.event_bus import EventBus
+from mindtrace.core.observables.observable_context import ObservableContext
+from mindtrace.core.types.task_schema import TaskSchema
+from mindtrace.core.utils.checks import check_libs, first_not_none, ifnone, ifnone_url
+from mindtrace.core.utils.dynamic import get_class, instantiate_target
+from mindtrace.core.utils.lambdas import named_lambda
+from mindtrace.core.utils.timers import Timeout, Timer, TimerCollection
+
+setup_logger()  # Initialize the default logger
+
+__all__ = [
+    "check_libs",
+    "ContextListener",
+    "Config",
+    "EventBus",
+    "first_not_none",
+    "get_class",
+    "ifnone",
+    "ifnone_url",
+    "instantiate_target",
+    "Mindtrace",
+    "MindtraceABC",
+    "MindtraceMeta",
+    "named_lambda",
+    "ObservableContext",
+    "TaskSchema",
+    "Timer",
+    "TimerCollection",
+    "Timeout",
+]

mindtrace-0.1.0/mindtrace/core/mindtrace/core/base/__init__.py

@@ -0,0 +1,7 @@
+from mindtrace.core.base.mindtrace_base import Mindtrace, MindtraceABC, MindtraceMeta
+
+__all__ = [
+    "Mindtrace",
+    "MindtraceABC",
+    "MindtraceMeta",
+]

mindtrace-0.1.0/mindtrace/core/mindtrace/core/base/mindtrace_base.py

@@ -0,0 +1,374 @@
+"""Mindtrace class. Provides unified configuration, logging and context management."""
+
+import inspect
+import logging
+import traceback
+from abc import ABC, ABCMeta
+from functools import wraps
+from typing import Callable, Optional
+
+from mindtrace.core.config import Config
+from mindtrace.core.logging.logger import get_logger
+from mindtrace.core.utils import ifnone
+
+
+class MindtraceMeta(type):
+    """Metaclass for Mindtrace class.
+
+    The MindtraceMeta metaclass enables classes deriving from Mindtrace to automatically use the same default logger within
+    class methods as it does within instance methods. I.e. consider the following class:
+
+    Example, logging in both class methods and instance methods::
+
+        from mindtrace.core import Mindtrace
+
+        class MyClass(Mindtrace):
+            def __init__(self):
+                super().__init__()
+
+            def instance_method(self):
+                self.logger.info(f"Using logger: {self.logger.name}")  # Using logger: mindtrace.my_module.MyClass
+
+            @classmethod
+            def class_method(cls):
+                cls.logger.info(f"Using logger: {cls.logger.name}")  # Using logger: mindtrace.my_module.MyClass
+    """
+
+    def __init__(cls, name, bases, attr_dict):
+        super().__init__(name, bases, attr_dict)
+        cls._logger = None
+
+    @property
+    def logger(cls):
+        if cls._logger is None:
+            cls._logger = get_logger(cls.unique_name)
+        return cls._logger
+
+    @logger.setter
+    def logger(cls, new_logger):
+        cls._logger = new_logger
+
+    @property
+    def unique_name(self) -> str:
+        return self.__module__ + "." + self.__name__
+
+
+class Mindtrace(metaclass=MindtraceMeta):
+    """Base class for all Mindtrace package core classes.
+
+    The Mindtrace class adds default context manager and logging methods. All classes that derive from Mindtrace can be
+    used as context managers and will use a unified logging format.
+
+    The class automatically provides logging capabilities for both class methods and instance methods.
+    For example:
+
+    .. code-block:: python
+
+        from mindtrace.core import Mindtrace
+
+        class MyClass(Mindtrace):
+            def __init__(self):
+                super().__init__()
+
+            def instance_method(self):
+                self.logger.info(f"Using logger: {self.logger.name}")  # Using logger: mindtrace.my_module.MyClass
+
+            @classmethod
+            def class_method(cls):
+                cls.logger.info(f"Using logger: {cls.logger.name}")  # Using logger: mindtrace.my_module.MyClass
+
+    The logging functionality is automatically provided through the MindtraceMeta metaclass,
+    which ensures consistent logging behavior across all method types.
+    """
+
+    config = Config()
+
+    def __init__(self, suppress: bool = False, **kwargs):
+        """
+        Initialize the Mindtrace object.
+
+        Args:
+            suppress (bool): Whether to suppress exceptions in context manager use.
+            **kwargs: Additional keyword arguments. Logger-related kwargs are passed to `get_logger`.
+                Valid logger kwargs: log_dir, logger_level, stream_level, file_level,
+                file_mode, propagate, max_bytes, backup_count
+        """
+        # Initialize parent classes first (cooperative inheritance)
+        try:
+            super().__init__(**kwargs)
+        except TypeError:
+            # If parent classes don't accept some kwargs, try without logger-specific ones
+            logger_param_names = {
+                "log_dir",
+                "logger_level",
+                "stream_level",
+                "file_level",
+                "file_mode",
+                "propagate",
+                "max_bytes",
+                "backup_count",
+            }
+            remaining_kwargs = {k: v for k, v in kwargs.items() if k not in logger_param_names}
+            try:
+                super().__init__(**remaining_kwargs)
+            except TypeError:
+                # If that still fails, try with no kwargs
+                super().__init__()
+
+        # Set Mindtrace-specific attributes
+        self.suppress = suppress
+
+        # Filter logger-specific kwargs for logger setup
+        logger_param_names = {
+            "log_dir",
+            "logger_level",
+            "stream_level",
+            "file_level",
+            "file_mode",
+            "propagate",
+            "max_bytes",
+            "backup_count",
+        }
+        logger_kwargs = {k: v for k, v in kwargs.items() if k in logger_param_names}
+
+        # Set up the logger
+        self.logger = get_logger(self.unique_name, **logger_kwargs)
+
+    @property
+    def unique_name(self) -> str:
+        return self.__module__ + "." + type(self).__name__
+
+    @property
+    def name(self) -> str:
+        return type(self).__name__
+
+    def __enter__(self):
+        self.logger.debug(f"Initializing {self.name} as a context manager.")
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self.logger.debug(f"Exiting context manager for {self.name}.")
+        if exc_type is not None:
+            info = (exc_type, exc_val, exc_tb)
+            self.logger.exception("Exception occurred", exc_info=info)
+            return self.suppress
+        return False
+
+    @classmethod
+    def autolog(
+        cls,
+        log_level=logging.DEBUG,
+        prefix_formatter: Optional[Callable] = None,
+        suffix_formatter: Optional[Callable] = None,
+        exception_formatter: Optional[Callable] = None,
+        self: Optional["Mindtrace"] = None,
+    ):
+        """Decorator that adds logger.log calls to the decorated method before and after the method is called.
+
+        By default, the autolog decorator will log the method name, arguments and keyword arguments before the method
+        is called, and the method name and result after the method completes. This behavior can be modified by passing
+        in prefix and suffix formatters.
+
+        The autolog decorator will also catch and log all Exceptions, re-raising any exception after logging it. The
+        behavior for autologging exceptions can be modified by passing in an exception_formatter.
+
+        The autolog decorator expects a logger to exist at self.logger, and hence can only be used by Mindtrace
+        subclasses or classes that have a logger attribute.
+
+        Args:
+            log_level: The log_level passed to logger.log().
+            prefix_formatter: The formatter used to log the command before the wrapped method runs. The prefix_formatter
+                will be given (and must accept) three arguments, in the following order:
+                - function: The function being wrapped.
+                - args: The args passed into the function.
+                - kwargs: The kwargs passed into the function.
+            suffix_formatter: The formatter used to log the command after the wrapped method runs. The suffix_formatter
+                will be given (and must accept) two arguments, in the following order:
+                - function: The function being wrapped.
+                - result: The result returned from the wrapped method.
+            exception_formatter: The formatter used to log any errors. The exception_formatter will be given (and must
+                accept) three arguments, in the following order:
+                - function: The function being wrapped.
+                - error: The caught Exception.
+                - stack trace: The stack trace, as provided by traceback.format_exc().
+            self: The instance of the class that the method is being called on. Self only needs to be passed in if the
+                wrapped method does not have self as the first argument. Refer to the example below for more details.
+
+        Example::
+
+            from mindtrace.core import Mindtrace
+
+            class MyClass(Mindtrace):
+                def __init__(self):
+                    super().__init__()
+
+                @Mindtrace.autolog()
+                def divide(self, arg1, arg2):
+                    self.logger.info("We are about to divide")
+                    result = arg1 / arg2
+                    self.logger.info("We have divided")
+                    return result
+
+            my_instance = MyClass()
+            my_instance.divide(1, 2)
+            my_instance.divide(1, 0)
+
+        The resulting log file should contain something similar to the following:
+
+        .. code-block:: text
+
+            MyClass - DEBUG - Calling divide with args: (1, 2) and kwargs: {}
+            MyClass - INFO - We are about to divide
+            MyClass - INFO - We have divided
+            MyClass - DEBUG - Finished divide with result: 0.5
+            MyClass - DEBUG - Calling divide with args: (1, 0) and kwargs: {}
+            MyClass - INFO - We are about to divide
+            MyClass - ERROR - division by zero
+            Traceback (most recent call last):
+            ...
+
+        If the wrapped method does not have self as the first argument, self must be passed in as an argument to the
+        autolog decorator.
+
+        .. code-block:: python
+
+            from fastapi import FastAPI
+            from mindtrace.core import Mindtrace
+
+            class MyClass(Mindtrace):
+                def __init__():
+                    super().__init__()
+
+                def create_app(self):
+                    app_ = FastAPI()
+
+                    @Mindtrace.autolog(self=self)  # self must be passed in as an argument as it is not captured in status()
+                    @app_.post("/status")
+                    def status():
+                        return {"status": "Available"}
+
+                    return app_
+
+        """
+        prefix_formatter = ifnone(
+            prefix_formatter,
+            default=lambda function,
+            args,
+            kwargs: f"Calling {function.__name__} with args: {args} and kwargs: {kwargs}",
+        )
+        suffix_formatter = ifnone(
+            suffix_formatter, default=lambda function, result: f"Finished {function.__name__} with result: {result}"
+        )
+        exception_formatter = ifnone(
+            exception_formatter,
+            default=lambda function,
+            e,
+            stack_trace: f"{function.__name__} failed to complete with the following error: {e}\n{stack_trace}",
+        )
+
+        def decorator(function):
+            is_async = inspect.iscoroutinefunction(function)
+
+            if self is None:
+                if is_async:
+
+                    @wraps(function)
+                    async def wrapper(self, *args, **kwargs):
+                        self.logger.log(log_level, prefix_formatter(function, args, kwargs))
+                        try:
+                            result = await function(self, *args, **kwargs)
+                        except Exception as e:
+                            self.logger.error(exception_formatter(function, e, traceback.format_exc()))
+                            raise
+                        else:
+                            self.logger.log(log_level, suffix_formatter(function, result))
+                            return result
+                else:
+
+                    @wraps(function)
+                    def wrapper(self, *args, **kwargs):
+                        self.logger.log(log_level, prefix_formatter(function, args, kwargs))
+                        try:
+                            result = function(self, *args, **kwargs)
+                        except Exception as e:
+                            self.logger.error(exception_formatter(function, e, traceback.format_exc()))
+                            raise
+                        else:
+                            self.logger.log(log_level, suffix_formatter(function, result))
+                            return result
+
+            else:
+                if is_async:
+
+                    @wraps(function)
+                    async def wrapper(*args, **kwargs):
+                        self.logger.log(log_level, prefix_formatter(function, args, kwargs))
+                        try:
+                            result = await function(*args, **kwargs)
+                        except Exception as e:
+                            self.logger.error(exception_formatter(function, e, traceback.format_exc()))
+                            raise
+                        else:
+                            self.logger.log(log_level, suffix_formatter(function, result))
+                            return result
+                else:
+
+                    @wraps(function)
+                    def wrapper(*args, **kwargs):
+                        self.logger.log(log_level, prefix_formatter(function, args, kwargs))
+                        try:
+                            result = function(*args, **kwargs)
+                        except Exception as e:
+                            self.logger.error(exception_formatter(function, e, traceback.format_exc()))
+                            raise
+                        else:
+                            self.logger.log(log_level, suffix_formatter(function, result))
+                            return result
+
+            return wrapper
+
+        return decorator
+
+
+class MindtraceABCMeta(MindtraceMeta, ABCMeta):
+    """Metaclass that combines MindtraceMeta and ABC metaclasses.
+
+    This metaclass resolves metaclass conflicts when creating classes that need to be both
+    abstract (using ABC) and have MindtraceMeta functionality. Python only allows a class to
+    have one metaclass, so this combined metaclass allows classes to inherit from both
+    Mindtrace class and ABC simultaneously.
+
+    Without this combined metaclass, trying to create a class that inherits from both Mindtrace class
+    and ABC would raise a metaclass conflict error since they each have different metaclasses.
+    """
+
+    pass
+
+
+class MindtraceABC(Mindtrace, ABC, metaclass=MindtraceABCMeta):
+    """Abstract base class combining Mindtrace class functionality with ABC support.
+
+    This class enables creating abstract classes that also have access to all Mindtrace features
+    such as logging, configuration, and context management. Use this class instead of
+    Mindtrace when you need to define abstract methods or properties in your class.
+
+    Example:
+        from mindtrace.core import MindtraceABC
+        from abc import abstractmethod
+
+        class MyAbstractService(MindtraceABC):
+            def __init__(self):
+                super().__init__()
+
+            @abstractmethod
+            def process_data(self, data):
+                '''Must be implemented by concrete subclasses.'''
+                pass
+
+    Note:
+        Without this class, attempting to create a class that inherits from both Mindtrace class and ABC
+        would fail due to metaclass conflicts. MindtraceABC resolves this by using the CombinedABCMeta.
+    """
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)

mindtrace-0.1.0/mindtrace/core/mindtrace/core/config/__init__.py

@@ -0,0 +1,3 @@
+from mindtrace.core.config.config import Config
+
+__all__ = ["Config"]

mindtrace-0.1.0/mindtrace/core/mindtrace/core/config/config.py

@@ -0,0 +1,27 @@
+import os
+
+
+class Config(dict):
+    """Template Config class."""
+
+    def __init__(self, **kwargs):
+        default_config = {
+            "MINDTRACE_TEMP_DIR": "~/.cache/mindtrace/temp",
+            "MINDTRACE_DEFAULT_REGISTRY_DIR": "~/.cache/mindtrace/registry",
+            "MINDTRACE_DEFAULT_HOST_URLS": {
+                "Service": "http://localhost:8000",
+            },
+            "MINDTRACE_MINIO_REGISTRY_URI": "~/.cache/mindtrace/minio-registry",
+            "MINDTRACE_MINIO_ENDPOINT": "localhost:9000",
+            "MINDTRACE_MINIO_ACCESS_KEY": "minioadmin",
+            "MINDTRACE_MINIO_SECRET_KEY": "minioadmin",
+            "MINDTRACE_SERVER_PIDS_DIR_PATH": "~/.cache/mindtrace/pids",
+            "MINDTRACE_LOGGER_DIR": "~/.cache/mindtrace/logs",
+        }
+        # Update defaults with any provided kwargs
+        default_config.update(kwargs)
+        # Expand ~ only if it is at the start of the string
+        for k, v in default_config.items():
+            if isinstance(v, str) and v.startswith("~/"):
+                default_config[k] = os.path.expanduser(v)
+        super().__init__(default_config)

mindtrace-0.1.0/mindtrace/core/mindtrace/core/logging/__init__.py

@@ -0,0 +1,3 @@
+from mindtrace.core.logging.logger import Logger
+
+__all__ = ["Logger"]