sciveo 0.1.59__tar.gz → 0.1.61__tar.gz

{sciveo-0.1.59 → sciveo-0.1.61}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: sciveo
-Version: 0.1.59
+Version: 0.1.61
 Description-Content-Type: text/markdown
 Provides-Extra: mon
 Provides-Extra: net
@@ -12,17 +12,90 @@ Provides-Extra: ml
 
 # SCIVEO - ML/AI and Scientific tools
 
+## Monitoring ML Infrastructure
+sciveo monitoring of (ML) machines' CPU/RAM/GPU/NET/DISK/TEMP etc.
+It is very easy to start and use with a single command line.
+
+ - **Monitoring** of current machine with sampling period of 60 seconds
+```shell
+sciveo monitor --period 60
+```
+ - **Monitoring** of current machine, logging the samples to local file (in this case the ./sciveo_monitor.json default name when dir path is used)
+```shell
+sciveo monitor --period 120 --output-path ./
+```
+
+The monitor client is sendind the monitored samples to remote [sciveo](https://sciveo.com) api (init config located in ~/.sciveo/)
+
+### Watchdog monitoring with action
+ - **Memory usage** watchdog for 90% max memory, measure every 10 seconds and execute "<some command to handle the high memory event>"
+```shell
+sciveo watchdog --src memory --threshold 90 --period 10 --execute "<some command to handle the high memory event>"
+```
+ - **Disk usage** watchdog for 80% max disk usage on /some/path/to/watch path, measure every 600 seconds and execute "<some command to handle the high disk usage event>".
+```shell
+sciveo watchdog --src disk --input-path "/some/path/to/watch" --threshold 80 --period 600 --execute "find /some/path/to/watch -type f -mtime +1"
+```
+
+
+## Network tools
+
+ - **Network Scan** Networks of hosts to scan for a single port
+ ```shell
+sciveo scan --port 22
+
+sciveo scan --net 192.168.0.0/24 --port 22 --timeout 0.5
+```
+
+- **Network Scan** Single Host list of ports to scan
+```shell
+sciveo scan --host 192.168.0.10
+
+sciveo scan --host 192.168.0.10 --ports [443,80]
+```
+
+## Remote network video recorder
+
+ - **NVR** RTSP cams video recorder
+ ```shell
+sciveo nvr --input-path cams.json
+```
+
+where cams.json has the cam and nvr configuration like
+```json
+{
+  "video_retention_period": 5,
+  "max_video_len": 60,
+  "transport": "tcp",
+  "path": {
+    "tmp": "tmp/nvr/tmp",
+    "video": "tmp/nvr/video"
+  },
+  "cam": {
+    "cam_livingroom": {
+      "url": "rtsp://192.168.1.11/stream1"
+    },
+    "cam_kitchen": {
+      "url": "rtsp://rtsp://192.168.1.12"
+    }
+  }
+}
+```
+
+
+## Media pipelines processing worker
+ - **Media Pipelines processing** service
+ ```shell
+sciveo media-server
+```
+
+
 ## Experiments Management Client
 `sciveo` is a Python library that serves as a client for managing machine learning and scientific experiments on the sciveo.com platform. This library provides a convenient interface to interact with the sciveo.com API, enabling users to organize, track, and analyze their experiments efficiently.
 There are few configuration params samplers, which allows easy parameter tuning. The "auto" sampler perhaps is the easiest to use, but also
 "random" and "grid" ones are available.
 
-## Monitoring client
-There is also the sciveo.monitor() which will start monitoring machine CPU/RAM/GPU/NET/DISK/TEMP etc.
-It is very easy to start and use.
-
-
-## Features
+#### Features
 
 - **Experiment Tracking:** Easily log and track your machine learning experiments.
 - **Experiment Comparison:** Compare different experiments and their results.
@@ -30,6 +103,7 @@ It is very easy to start and use.
 - **Integration with sciveo.com:** Seamlessly connect and synchronize with the sciveo.com platform.
 - **Monitoring machines (from HPC to jetson nano):** Visualisation and metrics collection in sciveo platform.
 
+
 ## Installation
 
  - main sciveo
@@ -39,10 +113,15 @@ pip install sciveo
 pip install sciveo[mon]
  - optional for sciveo network tools
 pip install sciveo[net]
-
- - for full installation
+ - for full installation, without media and ML support
 pip install sciveo[all]
 
+ - for media (images/video/audio/text)
+pip install sciveo[media]
+ - for media with ML processors
+pip install sciveo[media-ml]
+
+
 ## Example usage
 
 There are few public examples in sciveo.com.
@@ -78,11 +157,6 @@ sciveo.monitor(period=120, block=False)
 
 ```
 
-Sciveo Scan hosts port tool available
-```shell
-sciveo scan --port 22 --timeout 0.5
-```
-
 
 Experimental Projects management
 

{sciveo-0.1.59 → sciveo-0.1.61}/README.md

@@ -1,16 +1,89 @@
 # SCIVEO - ML/AI and Scientific tools
 
+## Monitoring ML Infrastructure
+sciveo monitoring of (ML) machines' CPU/RAM/GPU/NET/DISK/TEMP etc.
+It is very easy to start and use with a single command line.
+
+ - **Monitoring** of current machine with sampling period of 60 seconds
+```shell
+sciveo monitor --period 60
+```
+ - **Monitoring** of current machine, logging the samples to local file (in this case the ./sciveo_monitor.json default name when dir path is used)
+```shell
+sciveo monitor --period 120 --output-path ./
+```
+
+The monitor client is sendind the monitored samples to remote [sciveo](https://sciveo.com) api (init config located in ~/.sciveo/)
+
+### Watchdog monitoring with action
+ - **Memory usage** watchdog for 90% max memory, measure every 10 seconds and execute "<some command to handle the high memory event>"
+```shell
+sciveo watchdog --src memory --threshold 90 --period 10 --execute "<some command to handle the high memory event>"
+```
+ - **Disk usage** watchdog for 80% max disk usage on /some/path/to/watch path, measure every 600 seconds and execute "<some command to handle the high disk usage event>".
+```shell
+sciveo watchdog --src disk --input-path "/some/path/to/watch" --threshold 80 --period 600 --execute "find /some/path/to/watch -type f -mtime +1"
+```
+
+
+## Network tools
+
+ - **Network Scan** Networks of hosts to scan for a single port
+ ```shell
+sciveo scan --port 22
+
+sciveo scan --net 192.168.0.0/24 --port 22 --timeout 0.5
+```
+
+- **Network Scan** Single Host list of ports to scan
+```shell
+sciveo scan --host 192.168.0.10
+
+sciveo scan --host 192.168.0.10 --ports [443,80]
+```
+
+## Remote network video recorder
+
+ - **NVR** RTSP cams video recorder
+ ```shell
+sciveo nvr --input-path cams.json
+```
+
+where cams.json has the cam and nvr configuration like
+```json
+{
+  "video_retention_period": 5,
+  "max_video_len": 60,
+  "transport": "tcp",
+  "path": {
+    "tmp": "tmp/nvr/tmp",
+    "video": "tmp/nvr/video"
+  },
+  "cam": {
+    "cam_livingroom": {
+      "url": "rtsp://192.168.1.11/stream1"
+    },
+    "cam_kitchen": {
+      "url": "rtsp://rtsp://192.168.1.12"
+    }
+  }
+}
+```
+
+
+## Media pipelines processing worker
+ - **Media Pipelines processing** service
+ ```shell
+sciveo media-server
+```
+
+
 ## Experiments Management Client
 `sciveo` is a Python library that serves as a client for managing machine learning and scientific experiments on the sciveo.com platform. This library provides a convenient interface to interact with the sciveo.com API, enabling users to organize, track, and analyze their experiments efficiently.
 There are few configuration params samplers, which allows easy parameter tuning. The "auto" sampler perhaps is the easiest to use, but also
 "random" and "grid" ones are available.
 
-## Monitoring client
-There is also the sciveo.monitor() which will start monitoring machine CPU/RAM/GPU/NET/DISK/TEMP etc.
-It is very easy to start and use.
-
-
-## Features
+#### Features
 
 - **Experiment Tracking:** Easily log and track your machine learning experiments.
 - **Experiment Comparison:** Compare different experiments and their results.
@@ -18,6 +91,7 @@ It is very easy to start and use.
 - **Integration with sciveo.com:** Seamlessly connect and synchronize with the sciveo.com platform.
 - **Monitoring machines (from HPC to jetson nano):** Visualisation and metrics collection in sciveo platform.
 
+
 ## Installation
 
  - main sciveo
@@ -27,10 +101,15 @@ pip install sciveo
 pip install sciveo[mon]
  - optional for sciveo network tools
 pip install sciveo[net]
-
- - for full installation
+ - for full installation, without media and ML support
 pip install sciveo[all]
 
+ - for media (images/video/audio/text)
+pip install sciveo[media]
+ - for media with ML processors
+pip install sciveo[media-ml]
+
+
 ## Example usage
 
 There are few public examples in sciveo.com.
@@ -66,11 +145,6 @@ sciveo.monitor(period=120, block=False)
 
 ```
 
-Sciveo Scan hosts port tool available
-```shell
-sciveo scan --port 22 --timeout 0.5
-```
-
 
 Experimental Projects management
 

{sciveo-0.1.59 → sciveo-0.1.61}/sciveo/cli.py

@@ -51,8 +51,9 @@ def main():
   parser.add_argument('--src', type=str, default=None, help='Source')
   parser.add_argument('--dst', type=str, default=None, help='Destination')
   parser.add_argument('--value', type=float, help='Value')
-  parser.add_argument('--threshold', type=float, help='Threshold')
+  parser.add_argument('--threshold', type=float, default=None, help='Threshold')
   parser.add_argument('--execute', type=str, default=None, help='Execute command')
+  parser.add_argument('--pid', type=int, default=None, help='Process PID')
 
   args = parser.parse_args()
 
@@ -83,7 +84,7 @@ def main():
     else:
       info(f"init, [{base_path}] already there")
   elif args.command == 'nvr':
-    from sciveo.media.tools.nvr import VideoRecorder
+    from sciveo.media.capture.nvr import VideoRecorder
     VideoRecorder(args.input_path).start()
   elif args.command == 'media-server':
     from sciveo.media.pipelines.server import __START_SCIVEO_MEDIA_SERVER__
@@ -93,13 +94,17 @@ def main():
       from sciveo.media.pipelines.processors.audio.audio_extractor_process import plot_audio
       plot_audio(args.width, args.height, args.rate, args.input_path, args.output_path)
   elif args.command == 'watchdog':
-    from sciveo.monitoring.watchdog.base import MemoryWatchDogDaemon, DiskWatchDogDaemon
     daemons = []
     if args.execute is not None:
       if args.src is None or args.src.startswith("mem"):
+        from sciveo.monitoring.watchdog.base import MemoryWatchDogDaemon
         daemons.append(MemoryWatchDogDaemon(threshold_percent=args.threshold, period=args.period, command=args.execute))
       elif args.src.startswith("disk") and args.input_path is not None:
+        from sciveo.monitoring.watchdog.base import DiskWatchDogDaemon
         daemons.append(DiskWatchDogDaemon(path=args.input_path, threshold_percent=args.threshold, period=args.period, command=args.execute))
+      elif args.src.startswith("process"):
+        from sciveo.monitoring.watchdog.process import ProcessWatchDogDaemon
+        daemons.append(ProcessWatchDogDaemon(pid=args.pid, process_cmd=args.dst, period=args.period, command=args.execute, thread_inactive_threshold=args.threshold))
     for daemon in daemons:
       daemon.start()
     while(True):

sciveo-0.1.61/sciveo/media/capture/cam.py

@@ -0,0 +1,156 @@
+#
+# Pavlin Georgiev, Softel Labs
+#
+# This is a proprietary file and may not be copied,
+# distributed, or modified without express permission
+# from the owner. For licensing inquiries, please
+# contact pavlin@softel.bg.
+#
+# 2024
+#
+
+import time
+import numpy as np
+import cv2
+import base64
+import threading
+import mss
+
+from sciveo.tools.logger import *
+from sciveo.tools.daemon import DaemonBase
+from sciveo.tools.timers import FPSCounter
+
+
+class CaptureBase(DaemonBase):
+  def __init__(self, src, tag="capture", period=0):
+    super().__init__(period=period)
+    self.src = src
+    self.tag = tag
+    self.lock_frame = threading.Lock()
+    self.frame = None
+    self.fps = FPSCounter(period=10, tag=f"{self.tag} [{self.src}]")
+
+  def close(self):
+    pass
+
+  def read(self):
+    with self.lock_frame:
+      return self.frame
+
+  def read_buf(self):
+    frame = self.read()
+    ret, buffer = cv2.imencode('.jpg', frame)
+    return buffer.tobytes()
+
+  def read_frame(self):
+    pass
+
+  def loop(self):
+    with self.lock_frame:
+      self.read_frame()
+      self.fps.update()
+
+
+class CameraDaemon(CaptureBase):
+  def __init__(self, cam_id=0, period=0):
+    super().__init__(src=cam_id, tag="cam", period=period)
+    self.cap = cv2.VideoCapture(cam_id)
+    debug(cam_id, "warming...")
+    time.sleep(1)
+
+  def close(self):
+    self.cap.release()
+
+  def read_frame(self):
+    ret, self.frame = self.cap.read()
+
+
+class PredictorCameraDaemon(DaemonBase):
+  def __init__(self, args, cam, client):
+    super().__init__()
+    self.args = args
+    self.client = client
+    self.cam = cam
+    self.frame = None
+    self.fps = FPSCounter(period=10, tag="predictor")
+    self.lock_frame = threading.Lock()
+
+  def loop(self):
+    frame = self.cam.read()
+    image_base64 = self.client.image_encoded(image=frame)
+
+    if self.args.input_type == 0:
+      params = {
+        "predictor": self.args.predictor,
+        "compressed": self.args.compressed,
+        "X": [{
+          "messages": [{
+            "role": "user",
+            "content": [
+              {"type": "image"},
+              {"type": "text", "text": self.args.prompt},
+            ]
+          }],
+          "images": [image_base64]
+        }]
+      }
+    elif self.args.input_type == 1:
+      params = {
+        "predictor": self.args.predictor,
+        "compressed": self.args.compressed,
+        "X": [image_base64]
+      }
+
+    r = self.client.predict(params)
+    prediction = r[self.args.predictor][0]
+    info(prediction)
+    self.fps.update()
+
+    if isinstance(prediction, list) == False and isinstance(prediction, dict) == False:
+      cv2.putText(frame, prediction, (10, 100),
+        fontFace=cv2.FONT_HERSHEY_SIMPLEX, fontScale=1.3, color=(255, 255, 255), thickness=3
+      )
+
+    with self.lock_frame:
+      self.frame = frame
+
+  def read(self):
+    with self.lock_frame:
+      return self.frame
+
+
+class ScreenDaemon(CaptureBase):
+  def __init__(self, src=1, period=0, region=None):
+    super().__init__(src=src, tag="screen", period=period)
+    self.sct = mss.mss()
+    self.monitor = self.sct.monitors[src]
+
+    if region:
+      self.monitor = {
+        "left": region[0],
+        "top": region[1],
+        "width": region[2],
+        "height": region[3],
+      }
+      debug(f"Capturing screen region: {self.monitor}")
+    else:
+      self.monitor = self.sct.monitors[src]
+      debug(f"Capturing full monitor {src}: {self.monitor}")
+
+  def close(self):
+    self.sct.close()
+
+  def read_frame(self):
+    sct_img = self.sct.grab(self.monitor)
+    self.frame = np.array(sct_img)
+    self.frame = cv2.cvtColor(self.frame, cv2.COLOR_BGRA2BGR)
+
+
+if __name__ == '__main__':
+  cap = ScreenDaemon(src=1, region=[300,300, 800,800])
+  cap.start()
+  while(True):
+    frame = cap.read()
+    frame = cv2.resize(frame, (250, 250))
+    cv2.imshow("captured", frame)
+    cv2.waitKey(1)

{sciveo-0.1.59 → sciveo-0.1.61}/sciveo/monitoring/watchdog/base.py

@@ -19,23 +19,24 @@ from sciveo.tools.simple_counter import RunCounter
 
 
 class BaseWatchDogDaemon(DaemonBase):
-  def __init__(self, threshold_percent=90, period=5, command="echo '⚠️'", monitored="none"):
+  def __init__(self, threshold_percent=90, period=5, command="echo '⚠️'", monitored="none", monitor_value="usage"):
     super().__init__(period=period)
     self.threshold_percent = threshold_percent
     self.command = command
     self.used_percent = 0
     self.monitored = monitored
-    self.printer = RunCounter(60, lambda: debug(f"{self.monitored} usage: {self.used_percent}%", f"threshold: {self.threshold_percent}%"))
+    self.monitor_value = monitor_value
+    self.printer = RunCounter(60, lambda: debug(f"{self.monitored} {self.monitor_value}: {self.used_percent}%", f"threshold: {self.threshold_percent}%"))
 
-  def get_used_percent(self):
-    return 0.0
+  def value(self):
+    return {"percent": 0.0}
 
   def loop(self):
     current_value = self.value()
     self.used_percent = current_value["percent"]
     self.printer.run()
     if self.used_percent > self.threshold_percent:
-      warning(f"⚠️ {self.monitored} usage {self.used_percent}% exceeded {self.threshold_percent}%, executing command: {self.command}")
+      warning(f"⚠️ {self.monitored} {self.monitor_value} {self.used_percent}% exceeded {self.threshold_percent}%, executing command: {self.command}")
       subprocess.run(self.command, shell=True)
 
 

sciveo-0.1.61/sciveo/monitoring/watchdog/process.py

@@ -0,0 +1,104 @@
+#
+# Pavlin Georgiev, Softel Labs
+#
+# This is a proprietary file and may not be copied,
+# distributed, or modified without express permission
+# from the owner. For licensing inquiries, please
+# contact pavlin@softel.bg.
+#
+# 2025
+#
+
+import time
+import psutil
+
+from sciveo.tools.logger import *
+from sciveo.monitoring.watchdog.base import BaseWatchDogDaemon
+
+
+class ProcessWatchDogDaemon(BaseWatchDogDaemon):
+  def __init__(self, pid=None, process_cmd=None, period=5, command="echo '⚠️ process error!'", thread_inactive_threshold=None):
+    self.process_cmd = process_cmd
+    self.threads = {}
+    self.thread_inactive_threshold = thread_inactive_threshold
+    super().__init__(threshold_percent=10, period=period, command=command, monitored="Process", monitor_value="is dead")
+
+    self.this_pid = psutil.Process().pid
+    self.pids = self.get_pids_by_cmd()
+    if pid is not None:
+      self.pids.append(pid)
+
+  def get_pids_by_cmd(self):
+    matched_pids = []
+    if self.process_cmd is not None:
+      for proc in psutil.process_iter(attrs=["pid", "cmdline"]):
+        if proc.info["pid"] == self.this_pid:
+          continue
+        try:
+          cmdline = " ".join(proc.info["cmdline"])
+          if self.process_cmd in cmdline:
+            matched_pids.append(proc.info["pid"])
+        except Exception:
+          continue
+      info(f"processes matching [{self.process_cmd}]", matched_pids)
+    return matched_pids
+
+  def get_threads(self, process):
+    current_time = time.time()
+    for t in process.threads():
+      cpu_time = t.user_time + t.system_time
+      if t.id not in self.threads:
+        self.threads[t.id] = {
+          "id": t.id,
+          "start_at": current_time,
+          "last_active": current_time,
+          "last_cpu": cpu_time
+        }
+      else:
+        if cpu_time != self.threads[t.id]["last_cpu"]:
+          self.threads[t.id]["last_active"] = current_time
+        self.threads[t.id]["last_cpu"] = cpu_time
+
+  def check_threads(self, pid):
+    current_time = time.time()
+    for tid, thr in self.threads.items():
+      if current_time - thr["last_active"] > self.thread_inactive_threshold:
+        warning(f"⚠️ process[{pid}] thread[{tid}] inactive from {thr['last_active']}")
+
+  def process_value(self, pid):
+    try:
+      p = psutil.Process(pid)
+      if p.is_running():
+        if p.status() != psutil.STATUS_ZOMBIE:
+
+          if self.thread_inactive_threshold is not None:
+            try:
+              self.get_threads(p)
+              self.check_threads()
+            except psutil.NoSuchProcess as e:
+              exception(e, pid)
+
+          return 0
+        else:
+          return 50
+    except psutil.NoSuchProcess:
+      pass
+    return 100
+
+  def value(self):
+    pv = 0.0
+    for pid in self.pids:
+      pv += self.process_value(pid)
+    return {"percent": pv}
+
+
+if __name__ == '__main__':
+  daemons = [
+    ProcessWatchDogDaemon(pid=87443, thread_inactive_threshold=10),
+    ProcessWatchDogDaemon(process_cmd="sciveo monitor", thread_inactive_threshold=10),
+  ]
+  for d in daemons:
+    d.start()
+
+  while(True):
+    time.sleep(30)

sciveo-0.1.61/sciveo/version.py

@@ -0,0 +1,2 @@
+
+__version__ = '0.1.61'