flexmetric 0.1.3__tar.gz → 0.2.0__tar.gz

flexmetric-0.2.0/PKG-INFO

@@ -0,0 +1,212 @@
+Metadata-Version: 2.4
+Name: flexmetric
+Version: 0.2.0
+Summary: A flexible Prometheus exporter for commands, databases, functions, and scripts.
+Home-page: https://github.com/nikhillingadhal1999/custom_prometheus_agent
+Author: Nikhil Lingadhal
+License: MIT
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.7
+Description-Content-Type: text/markdown
+Requires-Dist: prometheus_client
+Requires-Dist: PyYAML
+Requires-Dist: psutil
+Requires-Dist: setuptools
+Requires-Dist: wheel
+Requires-Dist: twine
+Requires-Dist: flask
+Dynamic: author
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# FlexMetric
+
+FlexMetric is a lightweight, flexible, and extensible Prometheus exporter that allows you to expose system metrics, database query results, Python function outputs, and externally submitted metrics via an optional Flask API as Prometheus-compatible metrics—with minimal setup and maximum customization.
+
+---
+
+## Features
+
+- Run shell commands and expose the results as Prometheus metrics.
+- Execute SQL queries (e.g., SQLite) and monitor database statistics.
+- Automatically discover and expose Python function outputs as metrics.
+- Expose an optional **Flask API** (`/update_metric`) to receive external metrics dynamically.
+- Modular and easy to extend—add your own custom integrations.
+- Built-in Prometheus HTTP server (`/metrics`) with configurable port.
+
+---
+
+## Installation
+
+Install from PyPI:
+
+```bash
+pip install flexmetric
+```
+## Usage
+
+Run FlexMetric from the command line:
+
+```bash
+flexmetric --commands --commands-config commands.yaml --port 8000
+```
+
+## Available Modes
+
+FlexMetric supports multiple modes that can be used individually or combined to expose metrics:
+
+| Mode            | Description                                                            | Required Configuration File(s)           |
+|-----------------|------------------------------------------------------------------------|------------------------------------------|
+| `--commands`     | Runs system commands and exports outputs as Prometheus metrics.         | `commands.yaml`                          |
+| `--database`     | Executes SQL queries on databases and exports results.                 | `database.yaml` and `queries.yaml`       |
+| `--functions`    | Discovers and runs user-defined Python functions and exports outputs.  | `executable_functions.txt`               |
+| `--expose-api`   | Exposes a Flask API (`/update_metric`) to receive external metrics.     | *No configuration file required*         |
+### Example of Using Multiple Modes Together
+
+```bash
+flexmetric --commands --commands-config commands.yaml --database --database-config database.yaml --queries-config queries.yaml
+```
+
+## Configuration File Examples
+
+Below are example configurations for each supported mode.
+
+## Using the Flask API in FlexMetric
+
+To use the Flask API for submitting external metrics, you need to start the agent with the `--expose-api` flag along with the Flask host and port.
+
+### Start FlexMetric with Flask API
+
+```bash
+flexmetric --expose-api --flask-port <port> --flask-host <host> --metrics-port <metrics-port>
+```
+
+## Example: Running FlexMetric with Flask API
+
+To run FlexMetric with both Prometheus metrics and the Flask API enabled:
+
+```bash
+flexmetric --expose-api --flask-port 5000 --flask-host 0.0.0.0 --port 8000
+```
+
+Prometheus metrics exposed at:
+http://localhost:8000/metrics
+
+Flask API exposed at:
+http://localhost:5000/update_metric
+
+### Submitting a Metric to the Flask API
+```bash
+curl -X POST http://localhost:5000/update_metric \
+-H "Content-Type: application/json" \
+-d '{
+  "result": [
+    { "label": "cpu_usage", "value": 42.5 }
+  ],
+  "labels": ["cpu"]
+}'
+
+```
+
+### commands.yaml
+
+```yaml
+commands:
+  - name: disk_usage
+    command: df -h
+    label: path
+    timeout_seconds: 60
+```
+```yaml
+databases:
+  - name: mydb
+    db_type: sqlite
+    db_connection: /path/to/my.db
+````
+```yaml
+queries:
+  - name: user_count
+    db_type: sqlite
+    db_name: mydb
+    query: "SELECT COUNT(*) FROM users;"
+    label: table
+    label_value: users
+```
+executable_functions.txt 
+```
+function_name_1
+function_name_2
+```
+
+## Python Function Output Format
+
+When using the `--functions` mode, each Python function you define is expected to return a dictionary in the following format:
+
+```python
+{
+    'result': [
+        {'label': <label_or_labels>, 'value': <numeric_value>}
+    ],
+    'labels': [<label_name_1>]
+}
+```
+
+### Explanation:
+
+| Key     | Description                                                               |
+|--------|---------------------------------------------------------------------------|
+| `result` | A list of dictionaries, each containing a `label` and a corresponding numeric `value`. |
+| `labels` | A list of label names (used as Prometheus labels).                        |
+
+
+## Command-Line Options
+
+The following command-line options are available when running FlexMetric:
+
+| Option              | Description                                              | Default                    |
+|---------------------|----------------------------------------------------------|----------------------------|
+| `--port`             | Port for the Prometheus metrics server (`/metrics`)      | `8000`                     |
+| `--commands`         | Enable commands mode                                      |                            |
+| `--commands-config`  | Path to commands YAML file                                | `commands.yaml`            |
+| `--database`         | Enable database mode                                      |                            |
+| `--database-config`  | Path to database YAML file                                | `database.yaml`            |
+| `--queries-config`   | Path to queries YAML file                                 | `queries.yaml`             |
+| `--functions`        | Enable Python functions mode                              |                            |
+| `--functions-file`   | Path to functions file                                    | `executable_functions.txt` |
+| `--scripts`          | Enable shell scripts mode                                 |                            |
+| `--scripts-config`   | Path to scripts YAML file                                 | `scripts.yaml`             |
+| `--expose-api`       | Enable Flask API mode to receive external metrics         |                            |
+| `--flask-port`       | Port for the Flask API (`/update_metric`)                 | `5000`                     |
+| `--flask-host`       | Hostname for the Flask API                                | `0.0.0.0`                  |
+### Example Command:
+
+```bash
+flexmetric --commands --commands-config commands.yaml --port 8000
+```
+## Example Prometheus Output
+
+Once FlexMetric is running, the `/metrics` endpoint will expose metrics in the Prometheus format.
+
+Example output:
+```bash
+disk_usage_gauge{path="/"} 45.0
+```
+
+Each metric includes labels and numeric values that Prometheus can scrape and visualize.
+
+---
+
+## Future Enhancements
+
+The following features are planned or under consideration to improve FlexMetric:
+
+- Support for additional databases such as PostgreSQL and MySQL.
+- Enhanced support for more complex scripts and richer label extraction.

flexmetric-0.2.0/README.md

@@ -0,0 +1,183 @@
+# FlexMetric
+
+FlexMetric is a lightweight, flexible, and extensible Prometheus exporter that allows you to expose system metrics, database query results, Python function outputs, and externally submitted metrics via an optional Flask API as Prometheus-compatible metrics—with minimal setup and maximum customization.
+
+---
+
+## Features
+
+- Run shell commands and expose the results as Prometheus metrics.
+- Execute SQL queries (e.g., SQLite) and monitor database statistics.
+- Automatically discover and expose Python function outputs as metrics.
+- Expose an optional **Flask API** (`/update_metric`) to receive external metrics dynamically.
+- Modular and easy to extend—add your own custom integrations.
+- Built-in Prometheus HTTP server (`/metrics`) with configurable port.
+
+---
+
+## Installation
+
+Install from PyPI:
+
+```bash
+pip install flexmetric
+```
+## Usage
+
+Run FlexMetric from the command line:
+
+```bash
+flexmetric --commands --commands-config commands.yaml --port 8000
+```
+
+## Available Modes
+
+FlexMetric supports multiple modes that can be used individually or combined to expose metrics:
+
+| Mode            | Description                                                            | Required Configuration File(s)           |
+|-----------------|------------------------------------------------------------------------|------------------------------------------|
+| `--commands`     | Runs system commands and exports outputs as Prometheus metrics.         | `commands.yaml`                          |
+| `--database`     | Executes SQL queries on databases and exports results.                 | `database.yaml` and `queries.yaml`       |
+| `--functions`    | Discovers and runs user-defined Python functions and exports outputs.  | `executable_functions.txt`               |
+| `--expose-api`   | Exposes a Flask API (`/update_metric`) to receive external metrics.     | *No configuration file required*         |
+### Example of Using Multiple Modes Together
+
+```bash
+flexmetric --commands --commands-config commands.yaml --database --database-config database.yaml --queries-config queries.yaml
+```
+
+## Configuration File Examples
+
+Below are example configurations for each supported mode.
+
+## Using the Flask API in FlexMetric
+
+To use the Flask API for submitting external metrics, you need to start the agent with the `--expose-api` flag along with the Flask host and port.
+
+### Start FlexMetric with Flask API
+
+```bash
+flexmetric --expose-api --flask-port <port> --flask-host <host> --metrics-port <metrics-port>
+```
+
+## Example: Running FlexMetric with Flask API
+
+To run FlexMetric with both Prometheus metrics and the Flask API enabled:
+
+```bash
+flexmetric --expose-api --flask-port 5000 --flask-host 0.0.0.0 --port 8000
+```
+
+Prometheus metrics exposed at:
+http://localhost:8000/metrics
+
+Flask API exposed at:
+http://localhost:5000/update_metric
+
+### Submitting a Metric to the Flask API
+```bash
+curl -X POST http://localhost:5000/update_metric \
+-H "Content-Type: application/json" \
+-d '{
+  "result": [
+    { "label": "cpu_usage", "value": 42.5 }
+  ],
+  "labels": ["cpu"]
+}'
+
+```
+
+### commands.yaml
+
+```yaml
+commands:
+  - name: disk_usage
+    command: df -h
+    label: path
+    timeout_seconds: 60
+```
+```yaml
+databases:
+  - name: mydb
+    db_type: sqlite
+    db_connection: /path/to/my.db
+````
+```yaml
+queries:
+  - name: user_count
+    db_type: sqlite
+    db_name: mydb
+    query: "SELECT COUNT(*) FROM users;"
+    label: table
+    label_value: users
+```
+executable_functions.txt 
+```
+function_name_1
+function_name_2
+```
+
+## Python Function Output Format
+
+When using the `--functions` mode, each Python function you define is expected to return a dictionary in the following format:
+
+```python
+{
+    'result': [
+        {'label': <label_or_labels>, 'value': <numeric_value>}
+    ],
+    'labels': [<label_name_1>]
+}
+```
+
+### Explanation:
+
+| Key     | Description                                                               |
+|--------|---------------------------------------------------------------------------|
+| `result` | A list of dictionaries, each containing a `label` and a corresponding numeric `value`. |
+| `labels` | A list of label names (used as Prometheus labels).                        |
+
+
+## Command-Line Options
+
+The following command-line options are available when running FlexMetric:
+
+| Option              | Description                                              | Default                    |
+|---------------------|----------------------------------------------------------|----------------------------|
+| `--port`             | Port for the Prometheus metrics server (`/metrics`)      | `8000`                     |
+| `--commands`         | Enable commands mode                                      |                            |
+| `--commands-config`  | Path to commands YAML file                                | `commands.yaml`            |
+| `--database`         | Enable database mode                                      |                            |
+| `--database-config`  | Path to database YAML file                                | `database.yaml`            |
+| `--queries-config`   | Path to queries YAML file                                 | `queries.yaml`             |
+| `--functions`        | Enable Python functions mode                              |                            |
+| `--functions-file`   | Path to functions file                                    | `executable_functions.txt` |
+| `--scripts`          | Enable shell scripts mode                                 |                            |
+| `--scripts-config`   | Path to scripts YAML file                                 | `scripts.yaml`             |
+| `--expose-api`       | Enable Flask API mode to receive external metrics         |                            |
+| `--flask-port`       | Port for the Flask API (`/update_metric`)                 | `5000`                     |
+| `--flask-host`       | Hostname for the Flask API                                | `0.0.0.0`                  |
+### Example Command:
+
+```bash
+flexmetric --commands --commands-config commands.yaml --port 8000
+```
+## Example Prometheus Output
+
+Once FlexMetric is running, the `/metrics` endpoint will expose metrics in the Prometheus format.
+
+Example output:
+```bash
+disk_usage_gauge{path="/"} 45.0
+```
+
+Each metric includes labels and numeric values that Prometheus can scrape and visualize.
+
+---
+
+## Future Enhancements
+
+The following features are planned or under consideration to improve FlexMetric:
+
+- Support for additional databases such as PostgreSQL and MySQL.
+- Enhanced support for more complex scripts and richer label extraction.

{flexmetric-0.1.3 → flexmetric-0.2.0}/flexmetric/metric_process/database_processing.py

@@ -2,15 +2,20 @@ import yaml
 import sqlite3
 import re
 
+
 def read_yaml_file(file_path):
-    with open(file_path, 'r') as f:
+    with open(file_path, "r") as f:
         return yaml.safe_load(f)
 
+
 def get_database_config(databases, db_name):
     for db in databases:
-        if db['name'] == db_name:
+        if db["name"] == db_name:
             return db
-    raise ValueError(f"[ERROR] Database config for '{db_name}' not found in database.yaml.")
+    raise ValueError(
+        f"[ERROR] Database config for '{db_name}' not found in database.yaml."
+    )
+
 
 def execute_sqlite_query(db_path, query):
     try:
@@ -24,11 +29,13 @@ def execute_sqlite_query(db_path, query):
         print(f"[ERROR] SQLite query failed on {db_path}: {e}")
         return None
 
+
 def is_safe_query(query):
     # Remove leading spaces and brackets
     cleaned_query = query.strip().lower()
     # Match only queries that start with "select"
-    return re.match(r'^\(*\s*select', cleaned_query) is not None
+    return re.match(r"^\(*\s*select", cleaned_query) is not None
+
 
 def process_database_queries(queries_file, databases_file):
     # Get queries from queries file
@@ -36,24 +43,26 @@ def process_database_queries(queries_file, databases_file):
     # Get database from database file
     databases_config = read_yaml_file(databases_file)
 
-    commands = queries_config.get('commands', [])
-    databases = databases_config.get('databases', [])
+    commands = queries_config.get("commands", [])
+    databases = databases_config.get("databases", [])
 
     all_results = []
 
     for cmd in commands:
         try:
-            db_conf = get_database_config(databases, cmd['database'])
+            db_conf = get_database_config(databases, cmd["database"])
 
-            if db_conf['db_type'] != 'sqlite':
-                print(f"[WARN] Unsupported database type: {db_conf['db_type']} in command {cmd['name']}")
+            if db_conf["db_type"] != "sqlite":
+                print(
+                    f"[WARN] Unsupported database type: {db_conf['db_type']} in command {cmd['name']}"
+                )
                 continue
 
-            db_path = db_conf['db_connection']
-            query = cmd['query']
-            label = cmd['label']
-            label_value = cmd['label_value']
-            # check if query is safe 
+            db_path = db_conf["db_connection"]
+            query = cmd["query"]
+            label = cmd["label"]
+            label_value = cmd["label_value"]
+            # check if query is safe
             if is_safe_query(query):
                 value = execute_sqlite_query(db_path, query)
             else:
@@ -62,14 +71,18 @@ def process_database_queries(queries_file, databases_file):
 
             if value is not None:
                 result = {
-                    'result': [{'label': label_value, 'value': value}],
-                    'labels': [label]
+                    "result": [{"label": label_value, "value": value}],
+                    "labels": [label],
                 }
                 all_results.append(result)
             else:
-                print(f"[INFO] No result for command '{cmd['name']}' on database '{cmd['database']}'")
+                print(
+                    f"[INFO] No result for command '{cmd['name']}' on database '{cmd['database']}'"
+                )
 
         except Exception as e:
-            print(f"[ERROR] Processing command '{cmd.get('name', 'unknown')}' failed: {e}")
+            print(
+                f"[ERROR] Processing command '{cmd.get('name', 'unknown')}' failed: {e}"
+            )
 
-    return all_results
+    return all_results

flexmetric-0.2.0/flexmetric/metric_process/expiring_queue.py

@@ -0,0 +1,56 @@
+from collections import deque
+import threading
+import time
+import sched
+
+class ExpiringMetricQueue:
+    def __init__(self, expiry_seconds=60, cleanup_interval=5):
+        self.queue = deque()
+        self.expiry_seconds = expiry_seconds
+        self.cleanup_interval = cleanup_interval
+        self.lock = threading.Lock()
+        self.scheduler = sched.scheduler(time.time, time.sleep)
+        self._start_cleanup()
+    
+    def put(self, metric):
+        metric['timestamp'] = time.time()
+        with self.lock:
+            self.queue.append(metric)
+    
+    def pop_all(self):
+        with self.lock:
+            items = list(self.queue)
+            self.queue.clear()
+            cleaned_items = []
+            for item in items:
+                cleaned_item = {k: v for k, v in item.items() if k != 'timestamp'}
+                cleaned_items.append(cleaned_item)
+
+            return cleaned_items
+    
+    def _start_cleanup_thread(self):
+        thread = threading.Thread(target=self._cleanup, daemon=True)
+        thread.start()
+    
+    def _start_cleanup(self):
+        def scheduled_cleanup():
+            self._cleanup()
+            # Schedule next run
+            self.scheduler.enter(self.cleanup_interval, 1, scheduled_cleanup)
+
+        # Schedule first run immediately
+        self.scheduler.enter(0, 1, scheduled_cleanup)
+        threading.Thread(target=self.scheduler.run, daemon=True).start()
+
+    def _cleanup(self):
+        current_time = time.time()
+        with self.lock:
+            original_len = len(self.queue)
+            self.queue = deque(
+                [item for item in self.queue if current_time - item['timestamp'] <= self.expiry_seconds]
+            )
+            cleaned = original_len - len(self.queue)
+            if cleaned > 0:
+                print(f"[MetricQueue] Cleaned {cleaned} expired metrics")
+
+metric_queue = ExpiringMetricQueue(expiry_seconds=60, cleanup_interval=60)

flexmetric-0.2.0/flexmetric/metric_process/process_commands.py

@@ -0,0 +1,111 @@
+import subprocess
+import yaml
+import re
+from flexmetric.logging_module.logger import get_logger
+
+logger = get_logger(__name__)
+
+logger.info("prometheus is running") 
+
+def read_commands_from_yaml(file_path):
+    with open(file_path, 'r') as f:
+        config = yaml.safe_load(f)
+    return config.get('commands', [])
+
+def execute_command_with_timeout(command, timeout):
+    try:
+        result = subprocess.run(command, shell=True, capture_output=True, text=True, timeout=timeout)
+        if result.returncode != 0:
+            logger.info(f"Exception in running the command {command}")
+            return ''
+        return result.stdout.strip()
+    except subprocess.TimeoutExpired:
+        return ''
+    except Exception as ex:
+        logger.error(f"Exception : {ex}")
+        return ''
+
+def parse_command_output(raw_output, label_column, value_column, fixed_label_value):
+    result_list = []
+    lines = raw_output.strip().splitlines()
+
+    for line in lines:
+        parts = line.strip().split()
+        if not parts:
+            continue
+
+        if label_column == 'fixed':
+            label = fixed_label_value or 'label'
+        else:
+            try:
+                label = parts[label_column]
+            except IndexError:
+                label = 'unknown'
+
+        try:
+            raw_value = parts[value_column]
+            cleaned_value = re.sub(r'[^\d\.\-]', '', raw_value)
+            value = float(cleaned_value) if cleaned_value else 1
+        except (IndexError, ValueError):
+            value = 1
+
+        result_list.append({'label': label, 'value': value})
+
+    return result_list
+
+def process_single_command(cmd_info):
+    command = cmd_info['command']
+    label_name = cmd_info['label']
+    timeout = cmd_info.get('timeout_seconds', 30)
+    label_column = cmd_info.get('label_column', -1)
+    value_column = cmd_info.get('value_column', 0)
+    fixed_label_value = cmd_info.get('label_value')
+
+    raw_output = execute_command_with_timeout(command, timeout)
+    if raw_output == '':
+        logger.warning(f"No results for command {command}")
+        return None
+
+    result_list = parse_command_output(raw_output, label_column, value_column, fixed_label_value)
+
+    return {
+        'result': result_list,
+        'labels': [label_name]
+    }
+
+def is_command_safe(command):
+    blacklist = ['rm', 'reboot', 'shutdown', 'halt', 'poweroff', 'mkfs', 'dd']
+    for dangerous_cmd in blacklist:
+        if dangerous_cmd in command.split():
+            return False
+    return True
+
+def process_commands(config_file):
+    commands = read_commands_from_yaml(config_file)
+    all_results = []
+
+    for cmd_info in commands:
+        command = cmd_info.get('command', '')
+        if not command:
+            logger.error("Command is missing in the configuration.")
+            continue
+
+        if not is_command_safe(command):
+            logger.warning(f"Command '{command}' is not allowed and will not be executed.")
+            continue
+
+        try:
+            formatted_result = process_single_command(cmd_info)
+            if formatted_result:
+                all_results.append(formatted_result)
+        except KeyError as e:
+            logger.error(f"Missing key in command configuration: {e}. Command: {cmd_info}")
+        except Exception as e:
+            logger.error(f"An error occurred while processing command '{command}': {e}")
+
+    return all_results
+
+# # Example usage:
+# if __name__ == "__main__":
+#     results = process_commands('/Users/nlingadh/code/custom_prometheus_agent/src/commands.yaml')
+#     print(results)