flexmetric 0.1.4__tar.gz → 0.3.0__tar.gz

flexmetric-0.3.0/PKG-INFO

@@ -0,0 +1,243 @@
+Metadata-Version: 2.4
+Name: flexmetric
+Version: 0.3.0
+Summary: A secure flexible Prometheus exporter for commands, databases, functions, and scripts.
+Home-page: https://github.com/nikhillingadhal1999/custom_prometheus_agent
+Author: Nikhil Lingadhal
+License: MIT
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.7
+Description-Content-Type: text/markdown
+Requires-Dist: prometheus_client
+Requires-Dist: PyYAML
+Requires-Dist: psutil
+Requires-Dist: setuptools
+Requires-Dist: wheel
+Requires-Dist: twine
+Requires-Dist: flask
+Dynamic: author
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# FlexMetric
+
+FlexMetric is a lightweight, flexible, and extensible Prometheus exporter that allows you to securely expose system metrics, database query results, Python function outputs, and externally submitted metrics—via an optional Flask API with HTTPS support—as Prometheus-compatible metrics, all with minimal setup and maximum customization.
+
+---
+
+## Features
+
+- Run shell commands and expose the results as Prometheus metrics.  
+  ➔ **Harmful commands (e.g., file deletion, system shutdown) are blocked for safety.**
+- Execute SQL queries (e.g., SQLite) and monitor database statistics.  
+  ➔ **Potentially dangerous queries (e.g., `DROP`, `DELETE`, `TRUNCATE`) are not allowed.**
+- Automatically discover and expose Python function outputs as metrics.
+- Expose an optional **Flask API** (`/update_metric`) to receive external metrics dynamically.
+- Modular and easy to extend—add your own custom integrations.
+- Built-in Prometheus HTTP server (`/metrics`) with configurable port.
+- **Supports HTTPS** to securely expose both metrics and API endpoints.
+- **Input sanitization** is performed to ensure only safe commands and queries are executed.
+
+
+---
+
+## Installation
+
+Install from PyPI:
+
+```bash
+pip install flexmetric
+```
+## Usage
+
+Run FlexMetric from the command line:
+
+```bash
+flexmetric --commands --commands-config commands.yaml --port 8000
+```
+
+## Available Modes
+
+FlexMetric supports multiple modes that can be used individually or combined to expose metrics:
+
+| Mode            | Description                                                            | Required Configuration File(s)           |
+|-----------------|------------------------------------------------------------------------|------------------------------------------|
+| `--commands`     | Runs system commands and exports outputs as Prometheus metrics.         | `commands.yaml`                          |
+| `--database`     | Executes SQL queries on databases and exports results.                 | `database.yaml` and `queries.yaml`       |
+| `--functions`    | Discovers and runs user-defined Python functions and exports outputs.  | `executable_functions.txt`               |
+| `--expose-api`   | Exposes a Flask API (`/update_metric`) to receive external metrics.     | *No configuration file required*         |
+### Example of Using Multiple Modes Together
+
+```bash
+flexmetric --commands --commands-config commands.yaml --database --database-config database.yaml --queries-config queries.yaml
+```
+
+## Configuration File Examples
+
+Below are example configurations for each supported mode.
+
+## Using the Flask API in FlexMetric
+
+To use the Flask API for submitting external metrics, you need to start the agent with the `--expose-api` flag along with the Flask host and port.
+
+### Start FlexMetric with Flask API
+
+```bash
+flexmetric --expose-api --port <port> --host <host>
+```
+
+## Example: Running FlexMetric with Flask API
+
+To run FlexMetric with both Prometheus metrics and the Flask API enabled:
+
+```bash
+flexmetric --expose-api --port 5000 --host 0.0.0.0
+```
+
+Prometheus metrics exposed at:
+http://localhost:5000/metrics
+
+Flask API exposed at:
+http://localhost:5000/update_metric
+
+### Submitting a Metric to the Flask API
+```bash
+curl -X POST http://localhost:5000/update_metric \
+-H "Content-Type: application/json" \
+-d '{
+  "result": [
+    { "label": "cpu_usage", "value": 42.5 }
+  ],
+  "labels": ["cpu"]
+}'
+
+```
+
+### Using flex metrics in secure mode
+
+```bash
+flexmetric --port 5000 --host 0.0.0.0 --enable-https --ssl-cert=cert.pem --ssl-key=key.pem
+```
+Prometheus metrics exposed at:
+https://localhost:5000/metrics
+
+Flask API exposed at:
+https://localhost:5000/update_metric
+
+### Submitting a Metric to the Flask API
+```bash
+curl -k -X POST https://localhost:5000/update_metric \
+-H "Content-Type: application/json" \
+-d '{
+  "result": [
+    { "label": "cpu_usage", "value": 42.5 }
+  ],
+  "labels": ["cpu"]
+}'
+
+```
+
+### commands.yaml
+
+```yaml
+commands:
+  - name: disk_usage
+    command: df -h
+    label: path
+    timeout_seconds: 60
+```
+```yaml
+databases:
+  - name: mydb
+    db_type: sqlite
+    db_connection: /path/to/my.db
+````
+```yaml
+queries:
+  - name: user_count
+    db_type: sqlite
+    db_name: mydb
+    query: "SELECT COUNT(*) FROM users;"
+    label: table
+    label_value: users
+```
+executable_functions.txt 
+```
+function_name_1
+function_name_2
+```
+
+## Python Function Output Format
+
+When using the `--functions` mode, each Python function you define is expected to return a dictionary in the following format:
+
+```python
+{
+    'result': [
+        {'label': <label_or_labels>, 'value': <numeric_value>}
+    ],
+    'labels': [<label_name_1>]
+}
+```
+
+### Explanation:
+
+| Key     | Description                                                               |
+|--------|---------------------------------------------------------------------------|
+| `result` | A list of dictionaries, each containing a `label` and a corresponding numeric `value`. |
+| `labels` | A list of label names (used as Prometheus labels).                        |
+
+
+## Command-Line Options
+
+The following command-line options are available when running FlexMetric:
+
+| Option              | Description                                              | Default                    |
+|---------------------|----------------------------------------------------------|----------------------------|
+| `--port`             | Port for the Prometheus metrics server (`/metrics`)      | `8000`                     |
+| `--commands`         | Enable commands mode                                      |                            |
+| `--commands-config`  | Path to commands YAML file                                | `commands.yaml`            |
+| `--database`         | Enable database mode                                      |                            |
+| `--database-config`  | Path to database YAML file                                | `database.yaml`            |
+| `--queries-config`   | Path to queries YAML file                                 | `queries.yaml`             |
+| `--functions`        | Enable Python functions mode                              |                            |
+| `--functions-file`   | Path to functions file                                    | `executable_functions.txt` |
+| `--expose-api`       | Enable Flask API mode to receive external metrics         |                            |
+| `--flask-port`       | Port for the Flask API (`/update_metric`)                 | `5000`                     |
+| `--flask-host`       | Hostname for the Flask API                                | `0.0.0.0`                  |
+| `--enable-https`     | Enable HTTPS for the Flask API                            |                            |
+| `--ssl-cert`         | Path to SSL certificate file (`cert.pem`)                 |                            |
+| `--ssl-key`          | Path to SSL private key file (`key.pem`)                  |                            |
+
+### Example Command:
+
+```bash
+flexmetric --commands --commands-config commands.yaml --port 8000
+```
+## Example Prometheus Output
+
+Once FlexMetric is running, the `/metrics` endpoint will expose metrics in the Prometheus format.
+
+Example output:
+```bash
+disk_usage_gauge{path="/"} 45.0
+```
+
+Each metric includes labels and numeric values that Prometheus can scrape and visualize.
+
+---
+
+## Future Enhancements
+
+The following features are planned or under consideration to improve FlexMetric:
+
+- Support for additional databases such as PostgreSQL and MySQL.
+- Enhanced support for more complex scripts and richer label extraction.

flexmetric-0.3.0/README.md

@@ -0,0 +1,214 @@
+# FlexMetric
+
+FlexMetric is a lightweight, flexible, and extensible Prometheus exporter that allows you to securely expose system metrics, database query results, Python function outputs, and externally submitted metrics—via an optional Flask API with HTTPS support—as Prometheus-compatible metrics, all with minimal setup and maximum customization.
+
+---
+
+## Features
+
+- Run shell commands and expose the results as Prometheus metrics.  
+  ➔ **Harmful commands (e.g., file deletion, system shutdown) are blocked for safety.**
+- Execute SQL queries (e.g., SQLite) and monitor database statistics.  
+  ➔ **Potentially dangerous queries (e.g., `DROP`, `DELETE`, `TRUNCATE`) are not allowed.**
+- Automatically discover and expose Python function outputs as metrics.
+- Expose an optional **Flask API** (`/update_metric`) to receive external metrics dynamically.
+- Modular and easy to extend—add your own custom integrations.
+- Built-in Prometheus HTTP server (`/metrics`) with configurable port.
+- **Supports HTTPS** to securely expose both metrics and API endpoints.
+- **Input sanitization** is performed to ensure only safe commands and queries are executed.
+
+
+---
+
+## Installation
+
+Install from PyPI:
+
+```bash
+pip install flexmetric
+```
+## Usage
+
+Run FlexMetric from the command line:
+
+```bash
+flexmetric --commands --commands-config commands.yaml --port 8000
+```
+
+## Available Modes
+
+FlexMetric supports multiple modes that can be used individually or combined to expose metrics:
+
+| Mode            | Description                                                            | Required Configuration File(s)           |
+|-----------------|------------------------------------------------------------------------|------------------------------------------|
+| `--commands`     | Runs system commands and exports outputs as Prometheus metrics.         | `commands.yaml`                          |
+| `--database`     | Executes SQL queries on databases and exports results.                 | `database.yaml` and `queries.yaml`       |
+| `--functions`    | Discovers and runs user-defined Python functions and exports outputs.  | `executable_functions.txt`               |
+| `--expose-api`   | Exposes a Flask API (`/update_metric`) to receive external metrics.     | *No configuration file required*         |
+### Example of Using Multiple Modes Together
+
+```bash
+flexmetric --commands --commands-config commands.yaml --database --database-config database.yaml --queries-config queries.yaml
+```
+
+## Configuration File Examples
+
+Below are example configurations for each supported mode.
+
+## Using the Flask API in FlexMetric
+
+To use the Flask API for submitting external metrics, you need to start the agent with the `--expose-api` flag along with the Flask host and port.
+
+### Start FlexMetric with Flask API
+
+```bash
+flexmetric --expose-api --port <port> --host <host>
+```
+
+## Example: Running FlexMetric with Flask API
+
+To run FlexMetric with both Prometheus metrics and the Flask API enabled:
+
+```bash
+flexmetric --expose-api --port 5000 --host 0.0.0.0
+```
+
+Prometheus metrics exposed at:
+http://localhost:5000/metrics
+
+Flask API exposed at:
+http://localhost:5000/update_metric
+
+### Submitting a Metric to the Flask API
+```bash
+curl -X POST http://localhost:5000/update_metric \
+-H "Content-Type: application/json" \
+-d '{
+  "result": [
+    { "label": "cpu_usage", "value": 42.5 }
+  ],
+  "labels": ["cpu"]
+}'
+
+```
+
+### Using flex metrics in secure mode
+
+```bash
+flexmetric --port 5000 --host 0.0.0.0 --enable-https --ssl-cert=cert.pem --ssl-key=key.pem
+```
+Prometheus metrics exposed at:
+https://localhost:5000/metrics
+
+Flask API exposed at:
+https://localhost:5000/update_metric
+
+### Submitting a Metric to the Flask API
+```bash
+curl -k -X POST https://localhost:5000/update_metric \
+-H "Content-Type: application/json" \
+-d '{
+  "result": [
+    { "label": "cpu_usage", "value": 42.5 }
+  ],
+  "labels": ["cpu"]
+}'
+
+```
+
+### commands.yaml
+
+```yaml
+commands:
+  - name: disk_usage
+    command: df -h
+    label: path
+    timeout_seconds: 60
+```
+```yaml
+databases:
+  - name: mydb
+    db_type: sqlite
+    db_connection: /path/to/my.db
+````
+```yaml
+queries:
+  - name: user_count
+    db_type: sqlite
+    db_name: mydb
+    query: "SELECT COUNT(*) FROM users;"
+    label: table
+    label_value: users
+```
+executable_functions.txt 
+```
+function_name_1
+function_name_2
+```
+
+## Python Function Output Format
+
+When using the `--functions` mode, each Python function you define is expected to return a dictionary in the following format:
+
+```python
+{
+    'result': [
+        {'label': <label_or_labels>, 'value': <numeric_value>}
+    ],
+    'labels': [<label_name_1>]
+}
+```
+
+### Explanation:
+
+| Key     | Description                                                               |
+|--------|---------------------------------------------------------------------------|
+| `result` | A list of dictionaries, each containing a `label` and a corresponding numeric `value`. |
+| `labels` | A list of label names (used as Prometheus labels).                        |
+
+
+## Command-Line Options
+
+The following command-line options are available when running FlexMetric:
+
+| Option              | Description                                              | Default                    |
+|---------------------|----------------------------------------------------------|----------------------------|
+| `--port`             | Port for the Prometheus metrics server (`/metrics`)      | `8000`                     |
+| `--commands`         | Enable commands mode                                      |                            |
+| `--commands-config`  | Path to commands YAML file                                | `commands.yaml`            |
+| `--database`         | Enable database mode                                      |                            |
+| `--database-config`  | Path to database YAML file                                | `database.yaml`            |
+| `--queries-config`   | Path to queries YAML file                                 | `queries.yaml`             |
+| `--functions`        | Enable Python functions mode                              |                            |
+| `--functions-file`   | Path to functions file                                    | `executable_functions.txt` |
+| `--expose-api`       | Enable Flask API mode to receive external metrics         |                            |
+| `--flask-port`       | Port for the Flask API (`/update_metric`)                 | `5000`                     |
+| `--flask-host`       | Hostname for the Flask API                                | `0.0.0.0`                  |
+| `--enable-https`     | Enable HTTPS for the Flask API                            |                            |
+| `--ssl-cert`         | Path to SSL certificate file (`cert.pem`)                 |                            |
+| `--ssl-key`          | Path to SSL private key file (`key.pem`)                  |                            |
+
+### Example Command:
+
+```bash
+flexmetric --commands --commands-config commands.yaml --port 8000
+```
+## Example Prometheus Output
+
+Once FlexMetric is running, the `/metrics` endpoint will expose metrics in the Prometheus format.
+
+Example output:
+```bash
+disk_usage_gauge{path="/"} 45.0
+```
+
+Each metric includes labels and numeric values that Prometheus can scrape and visualize.
+
+---
+
+## Future Enhancements
+
+The following features are planned or under consideration to improve FlexMetric:
+
+- Support for additional databases such as PostgreSQL and MySQL.
+- Enhanced support for more complex scripts and richer label extraction.

{flexmetric-0.1.4 → flexmetric-0.3.0}/flexmetric/metric_process/database_processing.py

@@ -2,15 +2,20 @@ import yaml
 import sqlite3
 import re
 
+
 def read_yaml_file(file_path):
-    with open(file_path, 'r') as f:
+    with open(file_path, "r") as f:
         return yaml.safe_load(f)
 
+
 def get_database_config(databases, db_name):
     for db in databases:
-        if db['name'] == db_name:
+        if db["name"] == db_name:
             return db
-    raise ValueError(f"[ERROR] Database config for '{db_name}' not found in database.yaml.")
+    raise ValueError(
+        f"[ERROR] Database config for '{db_name}' not found in database.yaml."
+    )
+
 
 def execute_sqlite_query(db_path, query):
     try:
@@ -24,11 +29,13 @@ def execute_sqlite_query(db_path, query):
         print(f"[ERROR] SQLite query failed on {db_path}: {e}")
         return None
 
+
 def is_safe_query(query):
     # Remove leading spaces and brackets
     cleaned_query = query.strip().lower()
     # Match only queries that start with "select"
-    return re.match(r'^\(*\s*select', cleaned_query) is not None
+    return re.match(r"^\(*\s*select", cleaned_query) is not None
+
 
 def process_database_queries(queries_file, databases_file):
     # Get queries from queries file
@@ -36,24 +43,26 @@ def process_database_queries(queries_file, databases_file):
     # Get database from database file
     databases_config = read_yaml_file(databases_file)
 
-    commands = queries_config.get('commands', [])
-    databases = databases_config.get('databases', [])
+    commands = queries_config.get("commands", [])
+    databases = databases_config.get("databases", [])
 
     all_results = []
 
     for cmd in commands:
         try:
-            db_conf = get_database_config(databases, cmd['database'])
+            db_conf = get_database_config(databases, cmd["database"])
 
-            if db_conf['db_type'] != 'sqlite':
-                print(f"[WARN] Unsupported database type: {db_conf['db_type']} in command {cmd['name']}")
+            if db_conf["db_type"] != "sqlite":
+                print(
+                    f"[WARN] Unsupported database type: {db_conf['db_type']} in command {cmd['name']}"
+                )
                 continue
 
-            db_path = db_conf['db_connection']
-            query = cmd['query']
-            label = cmd['label']
-            label_value = cmd['label_value']
-            # check if query is safe 
+            db_path = db_conf["db_connection"]
+            query = cmd["query"]
+            label = cmd["label"]
+            label_value = cmd["label_value"]
+            # check if query is safe
             if is_safe_query(query):
                 value = execute_sqlite_query(db_path, query)
             else:
@@ -62,14 +71,18 @@ def process_database_queries(queries_file, databases_file):
 
             if value is not None:
                 result = {
-                    'result': [{'label': label_value, 'value': value}],
-                    'labels': [label]
+                    "result": [{"label": label_value, "value": value}],
+                    "labels": [label],
                 }
                 all_results.append(result)
             else:
-                print(f"[INFO] No result for command '{cmd['name']}' on database '{cmd['database']}'")
+                print(
+                    f"[INFO] No result for command '{cmd['name']}' on database '{cmd['database']}'"
+                )
 
         except Exception as e:
-            print(f"[ERROR] Processing command '{cmd.get('name', 'unknown')}' failed: {e}")
+            print(
+                f"[ERROR] Processing command '{cmd.get('name', 'unknown')}' failed: {e}"
+            )
 
-    return all_results
+    return all_results

flexmetric-0.3.0/flexmetric/metric_process/expiring_queue.py

@@ -0,0 +1,56 @@
+from collections import deque
+import threading
+import time
+import sched
+
+class ExpiringMetricQueue:
+    def __init__(self, expiry_seconds=60, cleanup_interval=5):
+        self.queue = deque()
+        self.expiry_seconds = expiry_seconds
+        self.cleanup_interval = cleanup_interval
+        self.lock = threading.Lock()
+        self.scheduler = sched.scheduler(time.time, time.sleep)
+        self._start_cleanup()
+    
+    def put(self, metric):
+        metric['timestamp'] = time.time()
+        with self.lock:
+            self.queue.append(metric)
+    
+    def pop_all(self):
+        with self.lock:
+            items = list(self.queue)
+            self.queue.clear()
+            cleaned_items = []
+            for item in items:
+                cleaned_item = {k: v for k, v in item.items() if k != 'timestamp'}
+                cleaned_items.append(cleaned_item)
+
+            return cleaned_items
+    
+    def _start_cleanup_thread(self):
+        thread = threading.Thread(target=self._cleanup, daemon=True)
+        thread.start()
+    
+    def _start_cleanup(self):
+        def scheduled_cleanup():
+            self._cleanup()
+            # Schedule next run
+            self.scheduler.enter(self.cleanup_interval, 1, scheduled_cleanup)
+
+        # Schedule first run immediately
+        self.scheduler.enter(0, 1, scheduled_cleanup)
+        threading.Thread(target=self.scheduler.run, daemon=True).start()
+
+    def _cleanup(self):
+        current_time = time.time()
+        with self.lock:
+            original_len = len(self.queue)
+            self.queue = deque(
+                [item for item in self.queue if current_time - item['timestamp'] <= self.expiry_seconds]
+            )
+            cleaned = original_len - len(self.queue)
+            if cleaned > 0:
+                print(f"[MetricQueue] Cleaned {cleaned} expired metrics")
+
+metric_queue = ExpiringMetricQueue(expiry_seconds=60, cleanup_interval=60)