sciveo 0.0.25__tar.gz → 0.1.89__tar.gz

{sciveo-0.0.25/sciveo.egg-info → sciveo-0.1.89}/PKG-INFO

@@ -1,26 +1,130 @@
 Metadata-Version: 2.1
 Name: sciveo
-Version: 0.0.25
+Version: 0.1.89
 Description-Content-Type: text/markdown
 Provides-Extra: mon
+Provides-Extra: net
+Provides-Extra: server
+Provides-Extra: media
+Provides-Extra: media-ml
+Provides-Extra: web
+Provides-Extra: db
+Provides-Extra: power
+Provides-Extra: all
+Provides-Extra: ml
+
+# SCIVEO - ML/AI and Scientific tools
+
+## Monitoring ML Infrastructure
+sciveo monitoring of (ML) machines' CPU/RAM/GPU/NET/DISK/TEMP etc.
+It is very easy to start and use with a single command line.
+
+ - **Monitoring** of current machine with sampling period of 60 seconds
+```shell
+sciveo monitor --period 60
+```
+ - **Monitoring** of current machine, logging the samples to local file (in this case the ./sciveo_monitor.json default name when dir path is used)
+```shell
+sciveo monitor --period 120 --output-path ./
+```
+
+The monitor client is sendind the monitored samples to remote [sciveo](https://sciveo.com) api (init config located in ~/.sciveo/)
+
+### Watchdog monitoring with action
+ - **Memory usage** watchdog for 90% max memory, measure every 10 seconds and execute "<some command to handle the high memory event>"
+```shell
+sciveo watchdog --src memory --threshold 90 --period 10 --execute "<some command to handle the high memory event>"
+```
+ - **Disk usage** watchdog for 80% max disk usage on /some/path/to/watch path, measure every 600 seconds and execute "<some command to handle the high disk usage event>".
+```shell
+sciveo watchdog --src disk --input-path "/some/path/to/watch" --threshold 80 --period 600 --execute "find /some/path/to/watch -type f -mtime +1"
+```
+
+
+## Network tools
+
+ - **Network Scan** Networks of hosts to scan for a single port
+ ```shell
+sciveo scan --port 22
+
+sciveo scan --net 192.168.0.0/24 --port 22 --timeout 0.5
+```
+
+- **Network Scan** Single Host list of ports to scan
+```shell
+sciveo scan --host 192.168.0.10
+
+sciveo scan --host 192.168.0.10 --ports [443,80]
+```
+
+## Remote network video recorder
+
+ - **NVR** RTSP cams video recorder
+ ```shell
+sciveo nvr --input-path cams.json
+```
+
+where cams.json has the cam and nvr configuration like
+```json
+{
+  "video_retention_period": 5,
+  "max_video_len": 60,
+  "transport": "tcp",
+  "path": {
+    "tmp": "tmp/nvr/tmp",
+    "video": "tmp/nvr/video"
+  },
+  "cam": {
+    "cam_livingroom": {
+      "url": "rtsp://192.168.1.11/stream1"
+    },
+    "cam_kitchen": {
+      "url": "rtsp://rtsp://192.168.1.12"
+    }
+  }
+}
+```
+
+
+## Media pipelines processing worker
+ - **Media Pipelines processing** service
+ ```shell
+sciveo media-server
+```
 
-# SCIVEO - ML/Scientific Experiments Management Client
 
+## Experiments Management Client
 `sciveo` is a Python library that serves as a client for managing machine learning and scientific experiments on the sciveo.com platform. This library provides a convenient interface to interact with the sciveo.com API, enabling users to organize, track, and analyze their experiments efficiently.
 There are few configuration params samplers, which allows easy parameter tuning. The "auto" sampler perhaps is the easiest to use, but also
 "random" and "grid" ones are available.
 
-## Features
+#### Features
 
 - **Experiment Tracking:** Easily log and track your machine learning experiments.
 - **Experiment Comparison:** Compare different experiments and their results.
 - **Data Visualization:** Visualize experiment metrics and results.
 - **Integration with sciveo.com:** Seamlessly connect and synchronize with the sciveo.com platform.
+- **Monitoring machines (from HPC to jetson nano):** Visualisation and metrics collection in sciveo platform.
+
 
 ## Installation
 
+ - main sciveo
 pip install sciveo
 
+ - optional for sciveo monitoring
+pip install sciveo[mon]
+ - optional for sciveo network tools
+pip install sciveo[net]
+ - for full installation, without media and ML support
+pip install sciveo[all]
+
+ - for media (images/video/audio/text)
+pip install sciveo[media]
+ - for media with ML processors
+pip install sciveo[media-ml]
+
+
 ## Example usage
 
 There are few public examples in sciveo.com.
@@ -28,8 +132,36 @@ There are few public examples in sciveo.com.
 The library has local and remote mode. The local one is ready to use, but for the remote one you will need a sciveo.com account.
 
 When have sciveo account:
+```shell
+sciveo init
+```
+Where ~/.sciveo/ path and ~/.sciveo/default file will be created. Just need to change the secret_access_key value.
+
+or
+```shell
 export SCIVEO_SECRET_ACCESS_KEY='my_sciveo_user_auth_token'
+```
+or create a file like ~/.sciveo/some_file_name where put:
+secret_access_key=my_sciveo_user_auth_token
+
+Sciveo Monitoring cli
+```shell
+sciveo monitor --period 120
+```
+
+Monitoring started along with other python code.
+```python
+import sciveo
+
+# Non blocking monitoring, so continue the code execution after it.
+sciveo.monitor(period=120, block=False)
+
+#rest of your python code here
+
+```
+
 
+Experimental Projects management
 
 ```python
 

sciveo-0.0.25/PKG-INFO → sciveo-0.1.89/README.md

@@ -1,26 +1,115 @@
-Metadata-Version: 2.1
-Name: sciveo
-Version: 0.0.25
-Description-Content-Type: text/markdown
-Provides-Extra: mon
+# SCIVEO - ML/AI and Scientific tools
 
-# SCIVEO - ML/Scientific Experiments Management Client
+## Monitoring ML Infrastructure
+sciveo monitoring of (ML) machines' CPU/RAM/GPU/NET/DISK/TEMP etc.
+It is very easy to start and use with a single command line.
 
+ - **Monitoring** of current machine with sampling period of 60 seconds
+```shell
+sciveo monitor --period 60
+```
+ - **Monitoring** of current machine, logging the samples to local file (in this case the ./sciveo_monitor.json default name when dir path is used)
+```shell
+sciveo monitor --period 120 --output-path ./
+```
+
+The monitor client is sendind the monitored samples to remote [sciveo](https://sciveo.com) api (init config located in ~/.sciveo/)
+
+### Watchdog monitoring with action
+ - **Memory usage** watchdog for 90% max memory, measure every 10 seconds and execute "<some command to handle the high memory event>"
+```shell
+sciveo watchdog --src memory --threshold 90 --period 10 --execute "<some command to handle the high memory event>"
+```
+ - **Disk usage** watchdog for 80% max disk usage on /some/path/to/watch path, measure every 600 seconds and execute "<some command to handle the high disk usage event>".
+```shell
+sciveo watchdog --src disk --input-path "/some/path/to/watch" --threshold 80 --period 600 --execute "find /some/path/to/watch -type f -mtime +1"
+```
+
+
+## Network tools
+
+ - **Network Scan** Networks of hosts to scan for a single port
+ ```shell
+sciveo scan --port 22
+
+sciveo scan --net 192.168.0.0/24 --port 22 --timeout 0.5
+```
+
+- **Network Scan** Single Host list of ports to scan
+```shell
+sciveo scan --host 192.168.0.10
+
+sciveo scan --host 192.168.0.10 --ports [443,80]
+```
+
+## Remote network video recorder
+
+ - **NVR** RTSP cams video recorder
+ ```shell
+sciveo nvr --input-path cams.json
+```
+
+where cams.json has the cam and nvr configuration like
+```json
+{
+  "video_retention_period": 5,
+  "max_video_len": 60,
+  "transport": "tcp",
+  "path": {
+    "tmp": "tmp/nvr/tmp",
+    "video": "tmp/nvr/video"
+  },
+  "cam": {
+    "cam_livingroom": {
+      "url": "rtsp://192.168.1.11/stream1"
+    },
+    "cam_kitchen": {
+      "url": "rtsp://rtsp://192.168.1.12"
+    }
+  }
+}
+```
+
+
+## Media pipelines processing worker
+ - **Media Pipelines processing** service
+ ```shell
+sciveo media-server
+```
+
+
+## Experiments Management Client
 `sciveo` is a Python library that serves as a client for managing machine learning and scientific experiments on the sciveo.com platform. This library provides a convenient interface to interact with the sciveo.com API, enabling users to organize, track, and analyze their experiments efficiently.
 There are few configuration params samplers, which allows easy parameter tuning. The "auto" sampler perhaps is the easiest to use, but also
 "random" and "grid" ones are available.
 
-## Features
+#### Features
 
 - **Experiment Tracking:** Easily log and track your machine learning experiments.
 - **Experiment Comparison:** Compare different experiments and their results.
 - **Data Visualization:** Visualize experiment metrics and results.
 - **Integration with sciveo.com:** Seamlessly connect and synchronize with the sciveo.com platform.
+- **Monitoring machines (from HPC to jetson nano):** Visualisation and metrics collection in sciveo platform.
+
 
 ## Installation
 
+ - main sciveo
 pip install sciveo
 
+ - optional for sciveo monitoring
+pip install sciveo[mon]
+ - optional for sciveo network tools
+pip install sciveo[net]
+ - for full installation, without media and ML support
+pip install sciveo[all]
+
+ - for media (images/video/audio/text)
+pip install sciveo[media]
+ - for media with ML processors
+pip install sciveo[media-ml]
+
+
 ## Example usage
 
 There are few public examples in sciveo.com.
@@ -28,8 +117,36 @@ There are few public examples in sciveo.com.
 The library has local and remote mode. The local one is ready to use, but for the remote one you will need a sciveo.com account.
 
 When have sciveo account:
+```shell
+sciveo init
+```
+Where ~/.sciveo/ path and ~/.sciveo/default file will be created. Just need to change the secret_access_key value.
+
+or
+```shell
 export SCIVEO_SECRET_ACCESS_KEY='my_sciveo_user_auth_token'
+```
+or create a file like ~/.sciveo/some_file_name where put:
+secret_access_key=my_sciveo_user_auth_token
+
+Sciveo Monitoring cli
+```shell
+sciveo monitor --period 120
+```
+
+Monitoring started along with other python code.
+```python
+import sciveo
+
+# Non blocking monitoring, so continue the code execution after it.
+sciveo.monitor(period=120, block=False)
+
+#rest of your python code here
+
+```
+
 
+Experimental Projects management
 
 ```python
 
@@ -139,4 +256,4 @@ sciveo.start(
 ### Who do I talk to? ###
 
 * Pavlin Georgiev
-* pavlin@softel.bg
+* pavlin@softel.bg

sciveo-0.1.89/sciveo/__init__.py

@@ -0,0 +1,56 @@
+#
+# Pavlin Georgiev, Softel Labs
+#
+# This is a proprietary file and may not be copied,
+# distributed, or modified without express permission
+# from the owner. For licensing inquiries, please
+# contact pavlin@softel.bg.
+#
+# 2023
+#
+
+from builtins import ImportError
+
+try:
+
+  import os
+  import time
+
+  from sciveo.tools.logger import *
+  from sciveo.tools.daemon import TasksDaemon, __upload_content__
+  from sciveo.content.runner import ProjectRunner
+  from sciveo.content.dataset import Dataset
+  from sciveo.version import __version__
+
+
+  TasksDaemon.current = TasksDaemon(num_threads=int(os.environ.get("SCIVEO_TASKS_NUM_THREADS", 1)))
+
+
+  # New Experiment
+  def open():
+    if ProjectRunner.current is not None:
+      return ProjectRunner.current.project
+    else:
+      error("there is no started project")
+
+  def start(project, function, configuration={}, **kwargs):
+    TasksDaemon.current.start()
+    ProjectRunner.current = ProjectRunner(project=project, function=function, configuration=configuration, **kwargs)
+    ProjectRunner.current.run()
+
+  # Dataset info
+  def dataset(info={}):
+    return Dataset.get(info)
+
+  # Monitoring start
+  def monitor(**kwargs):
+    from sciveo.monitoring.start import MonitorStart
+    MonitorStart(**kwargs)()
+
+  # Network tools
+  def network(**kwargs):
+    from sciveo.network.tools import NetworkTools
+    return NetworkTools(**kwargs)
+
+except ImportError as e:
+  pass

{sciveo-0.0.25 → sciveo-0.1.89}/sciveo/api/base.py

@@ -14,16 +14,19 @@ import json
 from urllib import request, parse
 from urllib.error import HTTPError
 
-from sciveo.common.tools.logger import *
+from sciveo.tools.logger import *
+from sciveo.tools.configuration import GlobalConfiguration
 
 
 class APIRemoteClient:
   def __init__(self, base_url=None, ver=1):
+    self.config = GlobalConfiguration.get()
+    self.auth_token = None
     if base_url is None:
-      base_url = os.environ.get("SCIVEO_API_BASE_URL", "https://sciveo.com")
+      base_url = self.config["api_base_url"]
     self.base_url = f"{base_url}/api/v{ver}/"
-    self.headers = { "Auth-Token": os.environ['SCIVEO_SECRET_ACCESS_KEY'] }
-    debug(type(self).__name__, f"base url: {self.base_url}")
+    self.headers = { "Auth-Token": self.config['secret_access_key'] }
+    debug(f"base url: {self.base_url}")
 
   def POST_SCI(self, content_type, data, timeout=30):
     return self.POST(f"sci/{content_type}/", data, timeout)
@@ -44,7 +47,7 @@ class APIRemoteClient:
     return result
 
   def GET(self, url, timeout=30):
-    url = f"{self.base_url}{url}&auth_token={os.environ['SCIVEO_SECRET_ACCESS_KEY']}"
+    url = f"{self.base_url}{url}&auth_token={self.config['secret_access_key']}"
     result = False
     try:
       # debug("GET", url)

sciveo-0.1.89/sciveo/api/predictors.py

@@ -0,0 +1,67 @@
+#
+# Pavlin Georgiev, Softel Labs
+#
+# This is a proprietary file and may not be copied,
+# distributed, or modified without express permission
+# from the owner. For licensing inquiries, please
+# contact pavlin@softel.bg.
+#
+# 2023
+#
+
+from sciveo.tools.logger import *
+from sciveo.tools.timers import Timer
+from sciveo.tools.configuration import GlobalConfiguration
+
+
+class Predictors:
+  instance = None
+
+  def __init__(self, predictors=None):
+    if predictors is not None:
+      self.list_predictors = predictors
+    else:
+      from sciveo.ml.images.embeddings import ImageEmbedding
+      from sciveo.ml.nlp.embeddings import TextEmbedding
+      self.list_predictors = {
+        "TextEmbedding": TextEmbedding(),
+        "ImageEmbedding": ImageEmbedding(),
+      }
+
+    self.stats = {}
+
+  @staticmethod
+  def get():
+    if Predictors.instance is None:
+      Predictors.instance = Predictors(GlobalConfiguration.get().data.get("API_PREDICTORS", None))
+    return Predictors.instance
+
+  @staticmethod
+  def set(predictors):
+    Predictors.instance = predictors
+
+  def predict(self, predictor_name, X):
+    t = Timer()
+    predicted = self.list_predictors[predictor_name].predict(X)
+    self.stats.setdefault(predictor_name, {"elapsed": 0.0, "count": 0})
+    self.stats[predictor_name]["elapsed"] += t.stop()
+    self.stats[predictor_name]["count"] += len(X)
+    return predicted
+
+  def train(self, predictor_name, X, Y_true=None):
+    t = Timer()
+    result = self.list_predictors[predictor_name].train(X, Y_true)
+    self.stats.setdefault(predictor_name, {"elapsed": 0.0, "count": 0})
+    self.stats[predictor_name]["elapsed"] += t.stop()
+    self.stats[predictor_name]["count"] += len(X)
+    return result
+
+  def describe(self):
+    result = []
+    for k, v in self.list_predictors.items():
+      result.append([k, v.describe(), self.stats.get(k, "")])
+    return result
+
+  def print(self):
+    for k, v in self.list_predictors.items():
+      info("predictor", k, v.describe())

sciveo-0.1.89/sciveo/api/server.py

@@ -0,0 +1,80 @@
+#
+# Pavlin Georgiev, Softel Labs
+#
+# This is a proprietary file and may not be copied,
+# distributed, or modified without express permission
+# from the owner. For licensing inquiries, please
+# contact pavlin@softel.bg.
+#
+# 2023
+#
+
+import os
+import time
+from flask import Flask, jsonify, render_template, request, url_for, Response, abort
+from waitress import serve
+
+from sciveo.tools.logger import *
+from sciveo.tools.daemon import DaemonBase
+from sciveo.tools.timers import Timer
+from sciveo.tools.compress import CompressJsonData
+from sciveo.tools.configuration import GlobalConfiguration
+from sciveo.api.predictors import Predictors
+
+
+CONFIG = GlobalConfiguration.get()
+SCI_API_AUTH_TOKEN = CONFIG["api_auth_token"]
+SCI_API_PREFIX = CONFIG["api_prefix"]
+
+
+app = Flask(__name__)
+
+
+def authenticate():
+  token = request.headers.get('Authorization')
+  if token != f"Bearer {SCI_API_AUTH_TOKEN}":
+    abort(401, 'Unauthorized')
+
+@app.before_request
+def before_request_func():
+  authenticate()
+
+@app.route(f'/{SCI_API_PREFIX}/predict', methods=['POST'])
+def predict_endpoint():
+  result = {"error": "invalid data"}
+  try:
+    if request.is_json:
+      data = request.json
+      if "predictor" in data and "X" in data:
+        t = Timer()
+        predicted = Predictors.get().predict(data["predictor"], data["X"])
+
+        if data.get("compressed", 0) > 0:
+          predicted = CompressJsonData().compress(predicted)
+
+        result = {
+          data["predictor"]: predicted,
+          "stats": {
+            "elapsed": t.stop(),
+            "count": len(data["X"])
+          }
+        }
+  except Exception as e:
+    exception("predict_endpoint", e)
+  return jsonify(result)
+
+
+class WebServerDaemon(DaemonBase):
+  def __init__(self, host="127.0.0.1", port=8901, predictors=None):
+    super().__init__()
+    self.host = host
+    self.port = port
+
+    if predictors is not None:
+      Predictors.set(predictors)
+
+  def run(self):
+    info("run", [self.host, self.port])
+    Predictors.get().print()
+
+    serve(app, host=self.host, port=self.port, threads=1, connection_limit=10)

{sciveo-0.0.25 → sciveo-0.1.89}/sciveo/api/upload.py

@@ -13,8 +13,9 @@ import os
 import json
 from urllib import request, parse
 from urllib.error import HTTPError
+import requests
 
-from sciveo.common.tools.logger import *
+from sciveo.tools.logger import *
 from sciveo.api.base import *
 
 
@@ -29,22 +30,20 @@ class APIFileUploader:
       APIFileUploader.api = APIRemoteClient()
 
   def __call__(self):
-    import requests
-
     file_name = self.local_path.split("/")[-1]
     url_presigned_post = f"aws/s3/presigned_post/?content_type={self.content_type}&file_name={file_name}"
     presigned_post = APIFileUploader.api.GET(url_presigned_post)
-    debug(type(self).__name__, "upload presigned_post", url_presigned_post, "=>", presigned_post)
+    debug("upload presigned_post", url_presigned_post, "=>", presigned_post)
     if "fields" in presigned_post:
       response = None
       with open(self.local_path, 'rb') as fh:
         files = { 'file': (presigned_post['fields']['key'], fh) }
         response = requests.post(presigned_post['url'], data=presigned_post['fields'], files=files)
       if response.status_code == 204:
-        debug(type(self).__name__, self.content_type, self.local_path, "uploaded")
+        debug(self.content_type, self.local_path, "uploaded")
         url_append = f"content/append/?content_type={self.content_type}&name={file_name}&key={presigned_post['fields']['key']}&parent_id={self.parent_guid}"
         result = APIFileUploader.api.GET(url_append)
-        debug(type(self).__name__, "upload", url_append, result, "appended")
+        debug("upload", url_append, result, "appended")
       else:
-        error(type(self).__name__, "upload", self.content_type, self.local_path, "FAIL")
+        error("upload", self.content_type, self.local_path, "FAIL")