cloudmesh-ai-common 7.0.4__tar.gz

cloudmesh_ai_common-7.0.4/PKG-INFO

@@ -0,0 +1,20 @@
+Metadata-Version: 2.4
+Name: cloudmesh-ai-common
+Version: 7.0.4
+Summary: CMC: Cloudmesh Command for AI
+License: Apache-2.0
+Keywords: cmc,extension,cli,click,cloudmesh
+Requires-Dist: click
+Requires-Dist: rich
+Requires-Dist: shellingham
+Requires-Dist: psutil
+Requires-Dist: PyYAML
+Requires-Dist: humanize
+Requires-Dist: tzlocal
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pip-tools; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+Requires-Dist: pytest-asyncio; extra == "dev"

cloudmesh_ai_common-7.0.4/README.md

@@ -0,0 +1,104 @@
+# Cloudmesh AI Common
+
+`cloudmesh-ai-common` provides a set of shared utilities for the Cloudmesh AI ecosystem, focusing on system introspection, structured telemetry, and standardized logging.
+
+## Features
+
+### 1. Enhanced System Introspection (`cloudmesh.ai.common.sys`)
+The system utility provides deep insights into the host hardware and environment.
+
+- **Hardware Detection**: 
+    - **GPU**: Automatic detection for NVIDIA, AMD, Apple Silicon, and Intel GPUs.
+    - **Topology**: Detection of NUMA nodes and high-speed network interfaces (InfiniBand/RoCE).
+    - **CPU**: Detailed processor descriptions across macOS, Windows, and Linux.
+- **Thermal Monitoring**: Standardized CPU temperature retrieval across Linux and macOS to detect thermal throttling.
+- **Container Awareness**: Automatic detection of Docker and Kubernetes environments, including extraction of K8s namespace metadata.
+- **Real-time Metrics**: 
+    - Live CPU utilization (overall and per-core).
+    - Detailed memory and swap usage.
+    - Disk I/O statistics and usage.
+- **System Info**: The `systeminfo()` function collects a comprehensive snapshot of the environment. Use `realtime=True` to include live performance metrics.
+
+### 2. Advanced Telemetry System (`cloudmesh.ai.common.telemetry`)
+A pluggable telemetry framework for emitting structured performance and status data from AI commands.
+
+- **Pluggable Backends**:
+    - `JSONFileBackend`: Emits records in JSONL format for easy ingestion by log aggregators.
+    - `SQLiteBackend`: Stores telemetry in a structured SQLite database for complex querying.
+    - `TextBackend`: Provides human-readable summaries for developers and operators.
+- **Automatic Tracking**: The `track()` context manager automatically handles start/complete/fail events and calculates task duration.
+- **Async Support**: The `AsyncTelemetry` class allows non-blocking telemetry emission using `asyncio`, ensuring that I/O operations do not interfere with compute-intensive AI tasks.
+- **Standardized Events**: Built-in helpers for `start()`, `complete()`, and `fail()` events.
+
+### 3. Telemetry Aggregation (`cloudmesh.ai.common.aggregation`)
+The `TelemetryAggregator` utility allows for the analysis of emitted telemetry data.
+
+- **Multi-source Loading**: Load data from either JSONL files or SQLite databases.
+- **Statistical Summaries**: Calculate success rates, status distributions, and command frequency.
+- **Metric Analysis**: Aggregate specific KPIs to find average, minimum, and maximum values across multiple runs.
+
+### 4. Standardized Logging (`cloudmesh.ai.common.logging`)
+Provides a consistent logging interface across all AI components to ensure uniform log formatting and traceability.
+
+## Usage Examples
+
+### System Info
+```python
+from cloudmesh.ai.common import sys as ai_sys
+
+# Get static and real-time system info
+info = ai_sys.systeminfo(realtime=True)
+print(f"CPU: {info['cpu']}")
+print(f"GPU Present: {info['gpu.present']}")
+print(f"CPU Temp: {info.get('cpu.temp')}")
+print(f"Container: {info.get('container.type', 'none')}")
+```
+
+### Synchronous Telemetry with Context Manager
+```python
+from cloudmesh.ai.common.telemetry import Telemetry, SQLiteBackend, TextBackend
+
+telemetry = Telemetry(
+    "my-ai-command", 
+    backends=[SQLiteBackend("metrics.db"), TextBackend()]
+)
+
+# Use the track context manager for automatic timing and error handling
+with telemetry.track(message="Running inference", metrics={"model": "llama-3"}):
+    # ... perform work ...
+    # If an exception occurs here, it is automatically logged as a 'failed' event
+    pass
+```
+
+### Asynchronous Telemetry
+```python
+import asyncio
+from cloudmesh.ai.common.telemetry import AsyncTelemetry, JSONFileBackend
+
+async def run_command():
+    telemetry = AsyncTelemetry(
+        "async-ai-task", 
+        backends=[JSONFileBackend("async_metrics.jsonl")]
+    )
+
+    async with telemetry.track(message="Processing async request"):
+        # ... perform async work ...
+        await asyncio.sleep(0.1)
+
+asyncio.run(run_command())
+```
+
+### Telemetry Aggregation
+```python
+from cloudmesh.ai.common.aggregation import TelemetryAggregator
+
+# Analyze data from a SQLite database
+agg = TelemetryAggregator("metrics.db")
+
+# Get high-level summary
+summary = agg.get_summary()
+print(f"Success Rate: {summary['success_rate']}")
+
+# Aggregate a specific metric across all runs
+latency_stats = agg.aggregate_metric("duration_sec")
+print(f"Average Duration: {latency_stats['avg']}s")

cloudmesh_ai_common-7.0.4/VERSION

@@ -0,0 +1 @@
+7.0.4

cloudmesh_ai_common-7.0.4/pyproject.toml

@@ -0,0 +1,55 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "cloudmesh-ai-common"
+dynamic = ["version"]
+description = "CMC: Cloudmesh Command for AI"
+license = {text = "Apache-2.0"}
+keywords = ["cmc", "extension", "cli", "click", "cloudmesh"]
+dependencies = [
+    "click",
+    "rich",
+    "shellingham",
+    "psutil",
+    "PyYAML",
+    "humanize",
+    "tzlocal",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest",
+    "pip-tools",
+    "twine",
+    "ruff",
+    "mypy",
+    "pytest-asyncio",
+]
+
+[tool.setuptools.dynamic]
+version = {file = ["VERSION"]}
+
+[tool.setuptools]
+# Tells setuptools to look inside the 'src' directory for packages
+package-dir = {"" = "src"}
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["cloudmesh*"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "N", "UP"]
+ignore = ["E501"]
+
+[tool.mypy]
+python_version = "3.10"
+strict = false
+ignore_missing_imports = true
+warn_return_any = true
+warn_unused_configs = true

cloudmesh_ai_common-7.0.4/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

cloudmesh_ai_common-7.0.4/src/cloudmesh/ai/common/__init__.py

@@ -0,0 +1,48 @@
+"""Common utilities for cloudmesh-ai components.
+
+This package provides shared functionality for logging, telemetry, system 
+information gathering, and other common helper utilities used across 
+the cloudmesh-ai ecosystem.
+"""
+
+from .logging import (
+    set_context_id,
+    get_context_id,
+    ContextFilter,
+    JsonFormatter,
+    load_logging_config,
+    get_log_dir,
+    ensure_log_dir,
+    get_log_file_path,
+    get_logger,
+    progress,
+)
+from .aggregation import TelemetryAggregator
+from .telemetry import (
+    TelemetryBackend,
+    JSONFileBackend,
+    SQLiteBackend,
+    TextBackend,
+    Telemetry,
+    AsyncTelemetry,
+)
+from .sys import (
+    os_is_windows,
+    os_is_mac,
+    os_is_linux,
+    os_is_pi,
+    has_window_manager,
+    sys_user,
+    get_platform,
+    get_cpu_description,
+    get_gpu_info,
+    get_thermal_info,
+    get_numa_info,
+    get_container_info,
+    get_network_info,
+    get_disk_read_speed,
+    get_cpu_metrics,
+    get_memory_metrics,
+    get_disk_metrics,
+    systeminfo,
+)

cloudmesh_ai_common-7.0.4/src/cloudmesh/ai/common/aggregation.py

@@ -0,0 +1,130 @@
+"""
+Telemetry aggregation utility for cloudmesh-ai.
+Provides tools to analyze and summarize telemetry data from various backends.
+"""
+
+import json
+import sqlite3
+from pathlib import Path
+from typing import Dict, Any, List, Optional, Union
+from collections import defaultdict
+
+class TelemetryAggregator:
+    """
+    Analyzes telemetry records to provide summaries and statistics.
+    Supports loading data from both JSONL files and SQLite databases.
+    """
+
+    def __init__(self, source: Union[str, Path]):
+        """
+        Initialize the TelemetryAggregator.
+
+        Args:
+            source: Path to the telemetry source (JSONL file or SQLite .db file).
+        """
+        self.source = Path(source).expanduser()
+        self.records: List[Dict[str, Any]] = []
+        self._load_data()
+
+    def _load_data(self) -> None:
+        """Detects source type and loads records into memory.
+        """
+        if self.source.suffix == ".db":
+            self._load_from_sqlite()
+        else:
+            self._load_from_jsonl()
+
+    def _load_from_jsonl(self) -> None:
+        """Loads records from a JSONL file.
+        """
+        try:
+            if not self.source.exists():
+                return
+            with open(self.source, "r") as f:
+                for line in f:
+                    if line.strip():
+                        self.records.append(json.loads(line))
+        except Exception as e:
+            print(f"Error loading JSONL telemetry: {e}")
+
+    def _load_from_sqlite(self) -> None:
+        """Loads records from a SQLite database.
+        """
+        try:
+            with sqlite3.connect(self.source) as conn:
+                conn.row_factory = sqlite3.Row
+                cursor = conn.execute("SELECT * FROM telemetry")
+                for row in cursor:
+                    record = dict(row)
+                    # Convert JSON strings back to dicts
+                    record["metrics"] = json.loads(record["metrics"]) if isinstance(record["metrics"], str) else record["metrics"]
+                    record["system"] = json.loads(record["system"]) if isinstance(record["system"], str) else record["system"]
+                    self.records.append(record)
+        except Exception as e:
+            print(f"Error loading SQLite telemetry: {e}")
+
+    def get_summary(self) -> Dict[str, Any]:
+        """
+        Calculates a high-level summary of the telemetry data.
+
+        Returns:
+            A dictionary containing total records, success rate, 
+            status distribution, and command distribution.
+        """
+        if not self.records:
+            return {"error": "No records found"}
+
+        total = len(self.records)
+        status_counts = defaultdict(int)
+        command_counts = defaultdict(int)
+        
+        for r in self.records:
+            status_counts[r.get("status", "unknown")] += 1
+            command_counts[r.get("command", "unknown")] += 1
+
+        success_count = status_counts.get("completed", 0)
+        
+        return {
+            "total_records": total,
+            "success_rate": f"{(success_count / total) * 100:.2f}%",
+            "status_distribution": dict(status_counts),
+            "command_distribution": dict(command_counts),
+        }
+
+    def aggregate_metric(self, metric_name: str) -> Dict[str, Any]:
+        """
+        Calculates average, min, and max for a specific metric across all records.
+
+        Args:
+            metric_name: The key of the metric to aggregate from the 'metrics' dictionary.
+
+        Returns:
+            A dictionary containing the count, average, minimum, and maximum values.
+        """
+        values = []
+        for r in self.records:
+            metrics = r.get("metrics", {})
+            if metric_name in metrics:
+                val = metrics[metric_name]
+                if isinstance(val, (int, float)):
+                    values.append(val)
+
+        if not values:
+            return {"metric": metric_name, "status": "no data"}
+
+        return {
+            "metric": metric_name,
+            "count": len(values),
+            "avg": sum(values) / len(values),
+            "min": min(values),
+            "max": max(values),
+        }
+
+if __name__ == "__main__":
+    # Simple test if run directly
+    import sys
+    if len(sys.argv) > 1:
+        agg = TelemetryAggregator(sys.argv[1])
+        print(json.dumps(agg.get_summary(), indent=4))
+    else:
+        print("Usage: python aggregation.py <telemetry_file_or_db>")

cloudmesh_ai_common-7.0.4/src/cloudmesh/ai/common/io.py

@@ -0,0 +1,101 @@
+"""
+I/O utility functions for cloudmesh-ai.
+Provides helpers for path expansion, YAML handling, and benchmark file creation.
+"""
+
+import os
+import yaml
+from pathlib import Path
+from typing import Any, Dict, Optional
+
+def path_expand(text: str, slashreplace: bool = True) -> str:
+    """Expands a path string by resolving '~', environment variables, and relative links.
+
+    Args:
+        text: The path to be expanded (e.g., "~/$PROJECT/./file.txt").
+        slashreplace: If True, returns backslashes on Windows. Defaults to True.
+
+    Returns:
+        The fully expanded and absolute path.
+    """
+    if not text:
+        return ""
+
+    # 1. Expand ~ and Environment Variables
+    expanded = os.path.expandvars(os.path.expanduser(text))
+    
+    # 2. Convert to a Path object and make it absolute
+    # .resolve() handles the "./" and "../" logic correctly
+    path_obj = Path(expanded).resolve()
+
+    # 3. Handle string conversion and slash preference
+    if slashreplace and os.name == 'nt':
+        # On Windows, this automatically uses backslashes
+        return str(path_obj)
+    
+    # .as_posix() forces forward slashes (/) regardless of OS
+    return path_obj.as_posix()
+
+def load_yaml(path: Path) -> Optional[Dict[str, Any]]:
+    """Safely loads a YAML file from the given path.
+
+    Args:
+        path: Path to the YAML file.
+
+    Returns:
+        The loaded YAML data as a dictionary, or None if the file does not exist 
+        or an error occurs.
+    """
+    try:
+        if not path.exists():
+            return None
+        with open(path, 'r') as f:
+            return yaml.safe_load(f)
+    except (yaml.YAMLError, OSError):
+        return None
+
+def dump_yaml(path: Path, data: Dict[str, Any]) -> None:
+    """Safely writes a dictionary to a YAML file, ensuring the directory exists.
+
+    Args:
+        path: Path where the YAML file should be written.
+        data: The dictionary to write to the file.
+
+    """
+    path.parent.mkdir(parents=True, exist_ok=True)
+    with open(path, 'w') as f:
+        yaml.dump(data, f, default_flow_style=False)
+
+def create_benchmark_yaml(path: str, n: int) -> None:
+    """Creates a Cloudmesh service YAML test file with specified number of services.
+
+    Args:
+        path: Path where the benchmark YAML file should be created.
+        n: Number of services to include in the benchmark file.
+
+    """
+    cm = {"cloudmesh": {}}
+    for i in range(0, n):
+        cm["cloudmesh"][f"service{i}"] = {"attribute": f"service{i}"}
+    
+    location = path_expand(path)
+    with open(location, "w") as yaml_file:
+        yaml.dump(cm, yaml_file, default_flow_style=False)
+
+def create_benchmark_file(path: str, n: int) -> int:
+    """Creates a file of a given size in binary megabytes.
+
+    Args:
+        path: Path where the benchmark file should be created.
+        n: Size of the file in megabytes.
+
+    Returns:
+        The actual size of the created file in megabytes.
+    """
+    location = path_expand(path)
+    size = 1048576 * n  # size in bytes
+    with open(location, "wb") as f:
+        f.write(os.urandom(size))
+    
+    s = os.path.getsize(location)
+    return int(s / 1048576.0)