argus-alm 0.12.3__tar.gz → 0.12.4b2__tar.gz

argus_alm-0.12.4b2/PKG-INFO

@@ -0,0 +1,129 @@
+Metadata-Version: 2.1
+Name: argus-alm
+Version: 0.12.4b2
+Summary: Argus
+Home-page: https://github.com/scylladb/argus
+License: Apache-2.0
+Author: Alexey Kartashov
+Author-email: alexey.kartashov@scylladb.com
+Requires-Python: >=3.10,<4.0
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: click (>=8.1.3,<9.0.0)
+Requires-Dist: requests (>=2.26.0,<3.0.0)
+Project-URL: Repository, https://github.com/scylladb/argus
+Description-Content-Type: text/markdown
+
+# Argus
+
+## Description
+
+Argus is a test tracking system intended to provide observability into automated test pipelines which use long-running resources. It allows observation of a test status, its events and its allocated resources. It also allows easy comparison between particular runs of a specific test.
+
+## Installation notes
+
+### Development
+
+For development setup instructions, see [dev-setup.md](./docs/dev-setup.md).
+
+### Prerequisites
+
+- Python >=3.10.0 (system-wide or pyenv)
+
+- NodeJS >=16 (with npm)
+
+- Yarn (can be installed globally with `npm -g install yarn`)
+
+- nginx
+
+- poetry >=1.2.0b1
+
+### From source
+
+#### Production
+
+Perform the following steps:
+
+Create a user that will be used by uwsgi:
+
+```bash
+useradd -m -s /bin/bash argus
+sudo -iu argus
+```
+
+(Optional) Install pyenv and create a virtualenv for this user:
+
+```bash
+pyenv install 3.10.0
+pyenv virtualenv argus
+pyenv activate argus
+```
+
+Clone the project into a directory somewhere where user has full write permissions
+
+```bash
+git clone https://github.com/scylladb/argus ~/app
+cd ~/app
+```
+
+Install project dependencies:
+
+```bash
+poetry install --with default,dev,web-backend,docker-image
+yarn install
+```
+
+Compile frontend files from `/frontend` into `/public/dist`
+
+```bash
+yarn webpack
+```
+
+Create a `argus.local.yaml` configuration file (used to configure database connection) and a `argus_web.yaml` (used for webapp secrets) in your application install directory.
+
+```bash
+cp argus_web.example.yaml argus_web.yaml
+cp argus.yaml argus.local.yaml
+```
+
+Open `argus.local.yaml` and add the database connection information (contact_points, user, password and keyspace name).
+
+Open `argus_web.yaml` and change the `SECRET_KEY` value to something secure, like a sha512 digest of random bytes. Fill out GITHUB_* variables with their respective values.
+
+Copy nginx configuration file from `docs/configs/argus.nginx.conf` to nginx virtual hosts directory:
+
+Ubuntu:
+
+```bash
+sudo cp docs/configs/argus.nginx.conf /etc/nginx/sites-available/argus
+sudo ln -s /etc/nginx/sites-enabled/argus /etc/nginx/sites-available/argus
+```
+
+RHEL/Centos/Alma/Fedora:
+
+```bash
+sudo cp docs/configs/argus.nginx.conf /etc/nginx/conf.d/argus.conf
+```
+
+Adjust the webhost settings in that file as necessary, particularly `listen` and `server_name` directives.
+
+Copy systemd service file from `docs/config/argus.service` to `/etc/systemd/system` directory:
+
+```bash
+sudo cp docs/config/argus.service /etc/systemd/system
+```
+
+Open it and adjust the path to the `start_argus.sh` script in the `ExecStart=` directive and the user/group, then reload systemd daemon configuration and enable (and optionally start) the service.
+
+WARNING: `start_argus.sh` assumes pyenv is installed into `~/.pyenv`
+
+```bash
+sudo systemctl daemon-reload
+sudo systemctl enable --now argus.service
+```
+
+
+

argus_alm-0.12.4b2/README.md

@@ -0,0 +1,109 @@
+# Argus
+
+## Description
+
+Argus is a test tracking system intended to provide observability into automated test pipelines which use long-running resources. It allows observation of a test status, its events and its allocated resources. It also allows easy comparison between particular runs of a specific test.
+
+## Installation notes
+
+### Development
+
+For development setup instructions, see [dev-setup.md](./docs/dev-setup.md).
+
+### Prerequisites
+
+- Python >=3.10.0 (system-wide or pyenv)
+
+- NodeJS >=16 (with npm)
+
+- Yarn (can be installed globally with `npm -g install yarn`)
+
+- nginx
+
+- poetry >=1.2.0b1
+
+### From source
+
+#### Production
+
+Perform the following steps:
+
+Create a user that will be used by uwsgi:
+
+```bash
+useradd -m -s /bin/bash argus
+sudo -iu argus
+```
+
+(Optional) Install pyenv and create a virtualenv for this user:
+
+```bash
+pyenv install 3.10.0
+pyenv virtualenv argus
+pyenv activate argus
+```
+
+Clone the project into a directory somewhere where user has full write permissions
+
+```bash
+git clone https://github.com/scylladb/argus ~/app
+cd ~/app
+```
+
+Install project dependencies:
+
+```bash
+poetry install --with default,dev,web-backend,docker-image
+yarn install
+```
+
+Compile frontend files from `/frontend` into `/public/dist`
+
+```bash
+yarn webpack
+```
+
+Create a `argus.local.yaml` configuration file (used to configure database connection) and a `argus_web.yaml` (used for webapp secrets) in your application install directory.
+
+```bash
+cp argus_web.example.yaml argus_web.yaml
+cp argus.yaml argus.local.yaml
+```
+
+Open `argus.local.yaml` and add the database connection information (contact_points, user, password and keyspace name).
+
+Open `argus_web.yaml` and change the `SECRET_KEY` value to something secure, like a sha512 digest of random bytes. Fill out GITHUB_* variables with their respective values.
+
+Copy nginx configuration file from `docs/configs/argus.nginx.conf` to nginx virtual hosts directory:
+
+Ubuntu:
+
+```bash
+sudo cp docs/configs/argus.nginx.conf /etc/nginx/sites-available/argus
+sudo ln -s /etc/nginx/sites-enabled/argus /etc/nginx/sites-available/argus
+```
+
+RHEL/Centos/Alma/Fedora:
+
+```bash
+sudo cp docs/configs/argus.nginx.conf /etc/nginx/conf.d/argus.conf
+```
+
+Adjust the webhost settings in that file as necessary, particularly `listen` and `server_name` directives.
+
+Copy systemd service file from `docs/config/argus.service` to `/etc/systemd/system` directory:
+
+```bash
+sudo cp docs/config/argus.service /etc/systemd/system
+```
+
+Open it and adjust the path to the `start_argus.sh` script in the `ExecStart=` directive and the user/group, then reload systemd daemon configuration and enable (and optionally start) the service.
+
+WARNING: `start_argus.sh` assumes pyenv is installed into `~/.pyenv`
+
+```bash
+sudo systemctl daemon-reload
+sudo systemctl enable --now argus.service
+```
+
+

{argus_alm-0.12.3 → argus_alm-0.12.4b2}/argus/backend/controller/admin_api.py

@@ -261,3 +261,29 @@ def get_tests_for_group():
         "status": "ok",
         "response": tests
     }
+
+
+@bp.route("/release/test/state/toggle", methods=["POST"])
+@check_roles(UserRoles.Admin)
+@api_login_required
+def quick_toggle_test_enabled():
+    
+    payload = get_payload(request)
+    res = ReleaseManagerService().toggle_test_enabled(test_id=payload["entityId"], new_state=payload["state"])
+    return {
+        "status": "ok",
+        "response": res
+    }
+
+
+@bp.route("/release/group/state/toggle", methods=["POST"])
+@check_roles(UserRoles.Admin)
+@api_login_required
+def quick_toggle_group_enabled():
+    
+    payload = get_payload(request)
+    res = ReleaseManagerService().toggle_group_enabled(group_id=payload["entityId"], new_state=payload["state"])
+    return {
+        "status": "ok",
+        "response": res
+    }

{argus_alm-0.12.3 → argus_alm-0.12.4b2}/argus/backend/controller/api.py

@@ -12,7 +12,8 @@ from argus.backend.controller.notification_api import bp as notifications_bp
 from argus.backend.controller.client_api import bp as client_bp
 from argus.backend.controller.testrun_api import bp as testrun_bp
 from argus.backend.controller.team import bp as team_bp
-from argus.backend.service.argus_service import ArgusService
+from argus.backend.controller.view_api import bp as view_bp
+from argus.backend.service.argus_service import ArgusService, ScheduleUpdateRequest
 from argus.backend.service.user import UserService, api_login_required
 from argus.backend.service.stats import ReleaseStatsCollector
 from argus.backend.models.web import ArgusRelease, ArgusGroup, ArgusTest, User, UserOauthToken
@@ -23,6 +24,7 @@ bp.register_blueprint(notifications_bp)
 bp.register_blueprint(client_bp)
 bp.register_blueprint(testrun_bp)
 bp.register_blueprint(team_bp)
+bp.register_blueprint(view_bp)
 bp.register_error_handler(Exception, handle_api_exception)
 LOGGER = logging.getLogger(__name__)
 
@@ -227,6 +229,8 @@ def release_schedules_submit():
         groups=payload["groups"],
         assignees=payload["assignees"],
         tag=payload["tag"],
+        comments=payload.get("comments"),
+        group_ids=payload.get("groupIds"),
     )
 
     return jsonify({
@@ -251,6 +255,27 @@ def release_schedules_delete():
     })
 
 
+@bp.route("/release/schedules/update", methods=["POST"])
+@api_login_required
+def release_schedule_update():
+    payload = get_payload(request)
+    req = ScheduleUpdateRequest(**payload)
+    service = ArgusService()
+    update_result = service.update_schedule(
+        release_id=req.release_id,
+        schedule_id=req.schedule_id,
+        old_tests=req.old_tests,
+        new_tests=req.new_tests,
+        comments=req.comments,
+        assignee=req.assignee
+    )
+
+    return jsonify({
+        "status": "ok",
+        "response": update_result
+    })
+
+
 @bp.route("/groups", methods=["GET"])
 @api_login_required
 def argus_groups():

{argus_alm-0.12.3 → argus_alm-0.12.4b2}/argus/backend/controller/main.py

@@ -1,3 +1,5 @@
+import datetime
+import json
 import logging
 from uuid import UUID
 from flask import (
@@ -8,6 +10,7 @@ from argus.backend.controller.team_ui import bp as teams_bp
 from argus.backend.service.argus_service import ArgusService
 from argus.backend.models.web import WebFileStorage
 from argus.backend.service.user import UserService, login_required
+from argus.backend.service.views import UserViewService
 
 LOGGER = logging.getLogger(__name__)
 
@@ -55,6 +58,24 @@ def releases():
     return render_template("releases.html.j2", releases=all_releases)
 
 
+@bp.route("/views")
+@login_required
+def views():
+    service = UserViewService()
+    all_views = service.get_all_views()
+    return render_template("views.html.j2", views=sorted(all_views, key=lambda view: view.created or datetime.datetime.fromtimestamp(0), reverse=True))
+
+
+@bp.route("/view/<string:view_name>")
+@login_required
+def view_dashboard(view_name: str):
+    service = UserViewService()
+    view = service.get_view_by_name(view_name=view_name)
+    data_json = view
+    view["widget_settings"] = json.loads(view["widget_settings"])
+    return render_template("view_dashboard.html.j2", data=data_json)
+
+
 @bp.route("/alert_debug")
 @login_required
 def alert_debug():

{argus_alm-0.12.3 → argus_alm-0.12.4b2}/argus/backend/controller/testrun_api.py

@@ -289,6 +289,22 @@ def ignore_jobs():
     }
 
 
+@bp.route("/get_runs_by_test_id_run_id", methods=["POST"])
+@api_login_required
+def get_runs_by_test_id_run_id():
+    payload: list[tuple[UUID, UUID]] = get_payload(request)
+    service = TestRunService()
+
+    result = service.resolve_run_build_id_and_number_multiple(payload)
+
+    return {
+        "status": "ok",
+        "response": {
+            "runs": result
+        }
+    }
+
+
 @bp.route("/jenkins/params", methods=["POST"])
 @api_login_required
 def get_jenkins_job_params():

argus_alm-0.12.4b2/argus/backend/controller/view_api.py

@@ -0,0 +1,162 @@
+import logging
+from uuid import UUID
+from flask import (
+    Blueprint,
+    jsonify,
+    request,
+)
+from argus.backend.error_handlers import handle_api_exception
+from argus.backend.models.web import User
+from argus.backend.service.stats import ViewStatsCollector
+from argus.backend.service.user import api_login_required
+from argus.backend.service.views import UserViewService
+from argus.backend.util.common import get_payload
+
+bp = Blueprint('view_api', __name__, url_prefix='/views')
+LOGGER = logging.getLogger(__name__)
+bp.register_error_handler(Exception, handle_api_exception)
+
+
+class ViewApiException(Exception):
+    pass
+
+
+@bp.route("/", methods=["GET"])
+@api_login_required
+def index():
+    return {
+        "status": "ok",
+        "response": {
+            "version": "v1",
+        }
+    }
+
+
+@bp.route("/create", methods=["POST"])
+@api_login_required
+def create_view():
+    payload = get_payload(request)
+    service = UserViewService()
+    view = service.create_view(
+        name=payload["name"], 
+        items=payload["items"],
+        widget_settings=payload["settings"],
+        description=payload.get("description"),
+        display_name=payload.get("displayName")
+    )
+    return {
+        "status": "ok",
+        "response": view
+    }
+
+
+@bp.route("/get", methods=["GET"])
+@api_login_required
+def get_view():
+    view_id = request.args.get("viewId")
+    if not view_id:
+        raise ViewApiException("No viewId provided.")
+    service = UserViewService()
+    view = service.get_view(UUID(view_id))
+    return {
+        "status": "ok",
+        "response": view
+    }
+
+
+@bp.route("/all", methods=["GET"])
+@api_login_required
+def get_all_views():
+    user_id = request.args.get("userId")
+    if user_id:
+        user = User.get(id=user_id)
+    else:
+        user = None
+    service = UserViewService()
+    views = service.get_all_views(user)
+    return {
+        "status": "ok",
+        "response": views
+    }
+
+
+@bp.route("/update", methods=["POST"])
+@api_login_required
+def update_view():
+    payload = get_payload(request)
+    service = UserViewService()
+    res = service.update_view(view_id=payload["viewId"], update_data=payload["updateData"])
+    return {
+        "status": "ok",
+        "response": res
+    }
+
+
+@bp.route("/delete", methods=["POST"])
+@api_login_required
+def delete_view():
+    payload = get_payload(request)
+    service = UserViewService()
+    res = service.delete_view(payload["viewId"])
+    return {
+        "status": "ok",
+        "response": res
+    }
+
+
+@bp.route("/search", methods=["GET"])
+@api_login_required
+def search_tests():
+    query = request.args.get("query")
+    service = UserViewService()
+    if query:
+        res = service.test_lookup(query)
+    else:
+        res = []
+    return {
+        "status": "ok",
+        "response": {
+            "hits": res,
+            "total": len(res)
+        }
+    }
+
+@bp.route("/stats", methods=["GET"])
+@api_login_required
+def view_stats():
+    view_id = request.args.get("viewId")
+    if not view_id:
+        raise ViewApiException("No view id provided.")
+    limited = bool(int(request.args.get("limited", 0)))
+    version = request.args.get("productVersion", None)
+    include_no_version = bool(int(request.args.get("includeNoVersion", True)))
+    force = bool(int(request.args.get("force",  0)))
+    collector = ViewStatsCollector(view_id=view_id, filter=version)
+    stats = collector.collect(limited=limited, force=force, include_no_version=include_no_version)
+
+    res = jsonify({
+        "status": "ok",
+        "response": stats
+    })
+    res.cache_control.max_age = 300
+    return res
+
+@bp.route("/<string:view_id>/versions", methods=["GET"])
+@api_login_required
+def view_versions(view_id: str):
+    service = UserViewService()
+    res = service.get_versions_for_view(view_id)
+    return {
+        "status": "ok",
+        "response": res
+    }
+
+@bp.route("/<string:view_id>/resolve", methods=["GET"])
+@api_login_required
+def view_resolve(view_id: str):
+    service = UserViewService()
+    res = service.resolve_view_for_edit(view_id)
+    return {
+        "status": "ok",
+        "response": res
+    }

{argus_alm-0.12.3 → argus_alm-0.12.4b2}/argus/backend/models/web.py

@@ -146,6 +146,20 @@ class ArgusGroup(Model):
             return super().__eq__(other)
 
 
+class ArgusUserView(Model):
+    id = columns.UUID(primary_key=True, partition_key=True, default=uuid4)
+    name = columns.Text(required=True, index=True)
+    display_name = columns.Text()
+    description = columns.Text()
+    user_id = columns.UUID(required=True, index=True)
+    tests = columns.List(value_type=columns.UUID, default=lambda: [])
+    release_ids = columns.List(value_type=columns.UUID, default=lambda: [])
+    group_ids = columns.List(value_type=columns.UUID, default=lambda: [])
+    created = columns.DateTime(default=datetime.utcnow)
+    last_updated = columns.DateTime(default=datetime.utcnow)
+    widget_settings = columns.Text(required=True)
+
+
 class ArgusTest(Model):
     __table_name__ = "argus_test_v2"
     id = columns.UUID(primary_key=True, default=uuid4)
@@ -180,6 +194,7 @@ class ArgusTestRunComment(Model):
     test_run_id = columns.UUID(required=True, index=True)
     user_id = columns.UUID(required=True, index=True)
     release_id = columns.UUID(required=True, index=True)
+    test_id = columns.UUID(required=True, index=True)
     posted_at = columns.Integer(
         required=True, clustering_order="desc", primary_key=True)
     message = columns.Text(min_length=1, max_length=65535)
@@ -350,6 +365,7 @@ USED_MODELS: list[Model] = [
     Team,
     WebFileStorage,
     ArgusRelease,
+    ArgusUserView,
     ArgusGroup,
     ArgusTest,
     ArgusTestRunComment,

{argus_alm-0.12.3 → argus_alm-0.12.4b2}/argus/backend/plugins/core.py

@@ -1,5 +1,6 @@
 import logging
 from datetime import datetime
+from math import ceil
 from uuid import UUID
 from time import time
 
@@ -16,8 +17,10 @@ from argus.backend.models.web import (
     ArgusSchedule,
     ArgusScheduleTest,
     ArgusScheduleAssignee,
+    ArgusUserView,
     User
 )
+from argus.backend.util.common import chunk
 from argus.backend.util.enums import TestInvestigationStatus, TestStatus
 
 LOGGER = logging.getLogger(__name__)
@@ -27,7 +30,7 @@ class PluginModelBase(Model):
     _plugin_name = "unknown"
     # Metadata
     build_id = columns.Text(required=True, partition_key=True)
-    start_time = columns.DateTime(required=True, primary_key=True, clustering_order="DESC", default=datetime.now, custom_index=True)
+    start_time = columns.DateTime(required=True, primary_key=True, clustering_order="DESC", default=datetime.utcnow, custom_index=True)
     id = columns.UUID(index=True, required=True)
     release_id = columns.UUID(index=True)
     group_id = columns.UUID(index=True)
@@ -136,17 +139,14 @@ class PluginModelBase(Model):
         cluster = ScyllaCluster.get()
         query = cluster.prepare(cls._stats_query())
         futures = []
-        step = 0
         step_size = 90
-        total_tests = len(build_ids)
-        while total_tests > 0:
-            id_slice = build_ids[step:step+step_size]
-            futures.append(cluster.session.execute_async(query=query, parameters=(id_slice,)))
-            step += step_size
-            total_tests = max(0, total_tests - step_size)
 
-        return futures
+        for step in range(0, ceil(len(build_ids) / step_size)):
+            start_pos = step*step_size
+            next_slice = build_ids[start_pos:start_pos+step_size]
+            futures.append(cluster.session.execute_async(query=query, parameters=(next_slice,)))
 
+        return futures
     @classmethod
     def get_run_meta_by_build_id(cls, build_id: str, limit: int = 10):
         cluster = ScyllaCluster.get()
@@ -177,6 +177,21 @@ class PluginModelBase(Model):
     def get_distinct_product_versions(cls, release: ArgusRelease) -> list[str]:
         raise NotImplementedError()
 
+    @classmethod
+    def get_distinct_versions_for_view(cls, tests: list[ArgusTest]) -> list[str]:
+        cluster = ScyllaCluster.get()
+        statement = cluster.prepare(f"SELECT scylla_version FROM {cls.table_name()} WHERE build_id IN ?")
+        futures = []
+        for batch in chunk(tests):
+            futures.append(cluster.session.execute_async(query=statement, parameters=([t.build_system_id for t in batch],)))
+        
+        rows = []
+        for future in futures:
+            rows.extend(future.result())
+        unique_versions = {r["scylla_version"] for r in rows if r["scylla_version"]}
+
+        return sorted(list(unique_versions), reverse=True)
+
     def update_heartbeat(self):
         self.heartbeat = int(time())
 
@@ -189,7 +204,7 @@ class PluginModelBase(Model):
     def submit_product_version(self, version: str):
         raise NotImplementedError()
 
-    def set_product_version(self, version: str):
+    def set_full_version(self, version: str):
         self.product_version = version
 
     def submit_logs(self, logs: list[dict]):