openqa-log-local 0.0.1__tar.gz

openqa_log_local-0.0.1/PKG-INFO

@@ -0,0 +1,121 @@
+Metadata-Version: 2.3
+Name: openqa-log-local
+Version: 0.0.1
+Summary: Create a local cache of openQA job log files
+Author: Michele Pagot
+Author-email: Michele Pagot <michele.pagot@suse.com>
+Requires-Dist: openqa-client>=4.3.1
+Requires-Dist: click
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+
+# openQA log local
+
+Library and cli to locally collect and inspect logs from openQA
+
+File will be locally cached on disk, downloaded and read transparently.
+
+## Dependency
+
+This package internally depend on [openQA-python-client](https://github.com/os-autoinst/openQA-python-client): please refer to
+documentation about openQA autentication.
+
+
+## Installation
+
+```bash
+pip install openqa_log_local
+```
+
+To install the package from the source code you can use `uv`:
+
+```bash
+uv pip install -e .
+```
+
+## Usage
+
+### Library
+
+To use the library in your Python project, you first need to import the `openQA_log_local` class:
+
+```python
+from openqa_log_local import openQA_log_local
+```
+
+Then, you can create an instance of the class, providing the openQA host URL:
+
+```python
+oll = openQA_log_local(host='http://openqa.opensuse.org')
+
+# Get job details
+log_details = oll.get_details(job_id=1234)
+
+# Get a list of log files associated to an openQA job.
+# No download any log file yet.
+log_list = oll.get_log_list(job_id=1234)
+log_txt_list = oll.get_log_list(job_id=4567, name_pattern=[r'*\.txt'])
+
+# Get content of a single log file. The file is downloaded to the cache
+# if not already available locally.
+# All the log file content is returned in `log_data`
+log_data = oll.get_log_data(job_id=1234, filename=log_list[3])
+
+# Get absolute path with filename of a single log file from the cache.
+# The file is downloaded to the cache if not already available locally.
+log_filename = oll.get_log_filename(job_id=1234, filename=log_list[3])
+```
+
+Cache can be configured:
+
+```python
+oll = openQA_log_local(
+    host='http://openqa.opensuse.org',
+    cache_location='/home/user/.openqa_cache',
+    max_size=100000,
+    time_to_live=3600)
+```
+
+Or also forced to be ignored and refreshed
+
+```python
+oll = openQA_log_local(
+    host='http://openqa.opensuse.org',
+    user_ignore_cache)
+```
+
+### CLI
+
+The package also provides a command-line interface (CLI) for interacting with openQA logs.
+
+#### Get Job Details
+
+```bash
+openqa-log-local get-details --host http://openqa.opensuse.org --job-id 1234
+```
+
+Run via `uv` if you have used `uv` to install it
+
+```bash
+uv run openqa-log-local get-details --host http://openqa.opensuse.org --job-id 1234
+```
+
+#### Get Log List
+
+```bash
+openqa-log-local get-log-list --host http://openqa.opensuse.org --job-id 1234
+```
+
+#### Get Log Data
+
+```bash
+openqa-log-local get-log-data --host http://openqa.opensuse.org --job-id 1234 --filename autoinst-log.txt
+```
+
+#### Get Log Filename
+
+```bash
+openqa-log-local get-log-filename --host http://openqa.opensuse.org --job-id 1234 --filename autoinst-log.txt
+```
+
+

openqa_log_local-0.0.1/README.md

@@ -0,0 +1,110 @@
+# openQA log local
+
+Library and cli to locally collect and inspect logs from openQA
+
+File will be locally cached on disk, downloaded and read transparently.
+
+## Dependency
+
+This package internally depend on [openQA-python-client](https://github.com/os-autoinst/openQA-python-client): please refer to
+documentation about openQA autentication.
+
+
+## Installation
+
+```bash
+pip install openqa_log_local
+```
+
+To install the package from the source code you can use `uv`:
+
+```bash
+uv pip install -e .
+```
+
+## Usage
+
+### Library
+
+To use the library in your Python project, you first need to import the `openQA_log_local` class:
+
+```python
+from openqa_log_local import openQA_log_local
+```
+
+Then, you can create an instance of the class, providing the openQA host URL:
+
+```python
+oll = openQA_log_local(host='http://openqa.opensuse.org')
+
+# Get job details
+log_details = oll.get_details(job_id=1234)
+
+# Get a list of log files associated to an openQA job.
+# No download any log file yet.
+log_list = oll.get_log_list(job_id=1234)
+log_txt_list = oll.get_log_list(job_id=4567, name_pattern=[r'*\.txt'])
+
+# Get content of a single log file. The file is downloaded to the cache
+# if not already available locally.
+# All the log file content is returned in `log_data`
+log_data = oll.get_log_data(job_id=1234, filename=log_list[3])
+
+# Get absolute path with filename of a single log file from the cache.
+# The file is downloaded to the cache if not already available locally.
+log_filename = oll.get_log_filename(job_id=1234, filename=log_list[3])
+```
+
+Cache can be configured:
+
+```python
+oll = openQA_log_local(
+    host='http://openqa.opensuse.org',
+    cache_location='/home/user/.openqa_cache',
+    max_size=100000,
+    time_to_live=3600)
+```
+
+Or also forced to be ignored and refreshed
+
+```python
+oll = openQA_log_local(
+    host='http://openqa.opensuse.org',
+    user_ignore_cache)
+```
+
+### CLI
+
+The package also provides a command-line interface (CLI) for interacting with openQA logs.
+
+#### Get Job Details
+
+```bash
+openqa-log-local get-details --host http://openqa.opensuse.org --job-id 1234
+```
+
+Run via `uv` if you have used `uv` to install it
+
+```bash
+uv run openqa-log-local get-details --host http://openqa.opensuse.org --job-id 1234
+```
+
+#### Get Log List
+
+```bash
+openqa-log-local get-log-list --host http://openqa.opensuse.org --job-id 1234
+```
+
+#### Get Log Data
+
+```bash
+openqa-log-local get-log-data --host http://openqa.opensuse.org --job-id 1234 --filename autoinst-log.txt
+```
+
+#### Get Log Filename
+
+```bash
+openqa-log-local get-log-filename --host http://openqa.opensuse.org --job-id 1234 --filename autoinst-log.txt
+```
+
+

openqa_log_local-0.0.1/pyproject.toml

@@ -0,0 +1,29 @@
+[project]
+name = "openqa-log-local"
+version = "0.0.1"
+description = "Create a local cache of openQA job log files"
+readme = "README.md"
+authors = [
+    { name = "Michele Pagot", email = "michele.pagot@suse.com" }
+]
+requires-python = ">=3.9"
+dependencies = [
+    "openqa-client>=4.3.1",
+    "click",
+]
+
+[project.scripts]
+openqa-log-local = "openqa_log_local.cli:cli"
+
+[build-system]
+requires = ["uv_build>=0.9.11,<0.10.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = [
+    "black>=25.11.0",
+    "mypy>=1.19.0",
+    "pytest>=8.4.2",
+    "ruff>=0.14.7",
+    "types-requests>=2.32.4.20250913",
+]

openqa_log_local-0.0.1/src/openqa_log_local/__init__.py

@@ -0,0 +1,4 @@
+from .cli import cli
+from .main import openQA_log_local
+
+__all__ = ["openQA_log_local", "cli"]

openqa_log_local-0.0.1/src/openqa_log_local/cache.py

@@ -0,0 +1,176 @@
+import os
+import logging
+import json
+from typing import Any, Optional
+
+
+class openQACache:
+    """Handles the file-based caching mechanism for openQA job data and logs.
+
+    This module provides the `openQACache` class, which is responsible
+    for storing and retrieving openQA job details and log files to and from
+    the backend local filesystem (usually your laptop).
+    The primary goal is to speed up analysis by avoiding repeated downloads
+    of the same data from the openQA server.
+
+    Architecture and Design
+    -----------------------
+
+    - **Directory Structure:** The cache is organized in a hierarchical structure.
+      A main cache directory (configurable by `cache_dir` in `config.yaml`)
+      contains subdirectories for each openQA server hostname.
+      Inside each hostname directory, cached data for a specific job is stored
+      in a JSON file named after the job ID (e.g., `.cache/openqa.suse.de/12345.json`).
+
+    - **Data Format:** Each cache file is a JSON object containing two main keys:
+      - `job_details`: A dictionary holding the complete JSON response for a job's
+        details from the openQA API.
+      - `log_files`: a list of log files downloaded from openQA and stored as
+        separated files assiciated to this job_id. Log files are stored in a folder
+        named with the value of the job_id, log filename is the one in this list.
+      - [DEPRECATED] `log_content`: A string containing the full content of the
+        `autoinst-log.txt` for that job.
+
+    - **Data Flow:** the API provided by this class are only responsible to manage
+                     openQA job details metadata and log file path.
+                     There is no API to write or read any log content.
+
+    Workflow
+    --------
+    The caching logic is integrated into the main application flow in `app/main.py`:
+
+    1.  **Job Discovery (`discover_jobs`):** When discovering related jobs, the
+        application first checks if a cache file exists for a given job ID using
+        `cache.is_details_cached()`. If it does, `cache.get_job_details()` is called
+        to retrieve the `job_details`, and the API call to the openQA server is skipped.
+
+    2.  **Log Processing (`process_job_logs`):** Before attempting to download a
+        log file, the application calls `cache.get_cached_log_filepath('filename.whatever')`.
+        If the log is found in the cache, the download is skipped.
+
+    3.  **Cache Writing (`_get_log_from_api`):** A cache file is written only after
+        job data and its corresponding log file have been successfully downloaded
+        from the openQA API. The `cache.write_data()` method is called to save
+        both the `job_details` and `log_content` into a single JSON file.
+
+    Configuration and Invalidation
+    ------------------------------
+    - The cache directory and maximum size are configured in the `config.yaml` file.
+    - As this project only consider and care about  completed jobs,
+      the cache never become invalid or obsolete due to changes in the openQA side.
+      Job details or log files are not supposed to change in the openQA server
+      for a completed jobs.
+    - The cache is persistent and does not have an automatic expiration or TTL
+      (Time To Live) mechanism. It can be manually cleared by deleting the cache
+      directory.
+    - The application frontend provides an `ignore_cache` option in the `/analyze`
+      endpoint to bypass the cache and force a fresh download of all data.
+      A user_ignore_cache is available in the class constructor. It allows to
+      annotate that the cache is there but user ask to ignore data from it.
+    """
+
+    def __init__(
+        self,
+        cache_path: str,
+        hostname: str,
+        max_size: int,
+        time_to_live: int,
+        logger: logging.Logger,
+    ) -> None:
+        """Initializes the cache handler.
+
+        Args:
+            cache_path (str): The root directory for the cache.
+            hostname (str): The openQA host, used to create a subdirectory in the cache.
+            max_size (int): The maximum size of the cache in bytes.
+            time_to_live (int): The time in seconds after which cached data is considered stale. -1 means
+                                        data never expires, 0 means data is
+                                        always refreshed.
+            logger (logging.Logger): The logger instance to use.
+        """
+        self.cache_path = cache_path
+        self.hostname = hostname
+        self.cache_host_dir = os.path.join(self.cache_path, self.hostname)
+        self.max_size = max_size
+        self.time_to_leave = time_to_live
+        self.logger = logger
+
+        os.makedirs(self.cache_path, exist_ok=True)
+        # os.makedirs(self.cache_host_dir, exist_ok=True)
+
+    def _file_path(self, job_id: str) -> str:
+        """Constructs the full path for a job's details metadata JSON file.
+
+        Args:
+            job_id (str): The ID of the job.
+
+        Returns:
+            str: The absolute path to the cache file for the job.
+        """
+        return os.path.join(self.cache_host_dir, f"{job_id}.json")
+
+    def is_details_cached(self, job_id: str) -> bool:
+        """Checks if the cache metadata file exists for a given job ID.
+
+        It also considers the time_to_live setting. If time_to_live is 0,
+        it will always return False.
+
+        Args:
+            job_id (str): The ID of the job.
+
+        Returns:
+            bool: True if the job details are in the cache and not stale, False otherwise.
+        """
+        if self.time_to_leave == 0:
+            return False
+        return os.path.exists(self._file_path(job_id))
+
+    def get_job_details(self, job_id: str) -> Optional[dict[str, Any]]:
+        """Retrieves cached job details for a specific job ID.
+
+        Args:
+            job_id (str): The ID of the job.
+
+        Returns:
+            Optional[dict[str, Any]]: A dictionary containing the job details,
+            or None if not found in cache or on error.
+        """
+        if self.time_to_leave == 0:
+            return None
+        try:
+            with open(self._file_path(job_id), "r") as f:
+                cached_data = json.load(f)
+                job_details: Optional[dict[str, Any]] = cached_data.get("job_details")
+                if job_details:
+                    return job_details
+                else:
+                    self.logger.info(
+                        f"Missing job_details in cached_data for job {job_id}"
+                    )
+                    return None
+        except (IOError, json.JSONDecodeError) as e:
+            self.logger.error(f"Error reading cache for job {job_id}: {e}")
+            return None
+
+    def write_details(
+        self, job_id: str, job_details: dict[str, Any], log_files: list[str]
+    ) -> None:
+        """Writes job details to a cache file.
+
+        Args:
+            job_id (str): The ID of the job.
+            job_details (dict[str, Any]): The dictionary of job details to cache.
+            log_files (list[str]): A list of log files associated with the job.
+        """
+        cache_file = self._file_path(job_id)
+        data_to_cache: dict[str, Any] = {
+            "job_details": job_details,
+        }
+
+        try:
+            os.makedirs(self.cache_host_dir, exist_ok=True)
+            with open(cache_file, "w") as f:
+                json.dump(data_to_cache, f)
+            self.logger.info(f"Successfully cached metadata for job {job_id}.")
+        except (IOError, TypeError) as e:
+            self.logger.error(f"Failed to write cache for job {job_id}: {e}")

openqa_log_local-0.0.1/src/openqa_log_local/cli.py

@@ -0,0 +1,84 @@
+import click
+import logging
+from .main import openQA_log_local
+
+
+@click.group()
+@click.option("--debug/--no-debug", default=False)
+@click.pass_context
+def cli(ctx, debug):
+    """A CLI to locally collect and inspect logs from openQA.
+
+    Files will be locally cached on disk, downloaded and read transparently.
+    """
+    ctx.ensure_object(dict)
+    ctx.obj["DEBUG"] = debug
+    if debug:
+        logging.basicConfig(level=logging.DEBUG)
+    else:
+        logging.basicConfig(level=logging.WARNING)
+
+
+@cli.command()
+@click.option("--host", required=True, help="The openQA host URL.")
+@click.option("--job-id", required=True, type=int, help="The job ID.")
+@click.pass_context
+def get_details(ctx, host, job_id):
+    """Get job details for a specific openQA job."""
+    oll = openQA_log_local(host=host)
+    details = oll.get_details(job_id)
+    if details is None:
+        click.echo(f"Job {job_id} not found.", err=True)
+        ctx.exit(1)
+    click.echo(details)
+
+
+@cli.command()
+@click.option("--host", required=True, help="The openQA host URL.")
+@click.option("--job-id", required=True, type=int, help="The job ID.")
+@click.option("--name-pattern", help="A regex pattern to filter log files.")
+@click.pass_context
+def get_log_list(ctx, host, job_id, name_pattern):
+    """Get a list of log files associated to an openQA job.
+
+    This command does not download any log file.
+    """
+    oll = openQA_log_local(host=host)
+    log_list = oll.get_log_list(job_id, name_pattern)
+    for log in log_list:
+        click.echo(log)
+
+
+@cli.command()
+@click.option("--host", required=True, help="The openQA host URL.")
+@click.option("--job-id", required=True, type=int, help="The job ID.")
+@click.option("--filename", required=True, help="The name of the log file.")
+@click.pass_context
+def get_log_data(ctx, host, job_id, filename):
+    """Get content of a single log file.
+
+    The file is downloaded to the cache if not already available locally.
+    All the log file content is returned.
+    """
+    oll = openQA_log_local(host=host)
+    log_data = oll.get_log_data(job_id, filename)
+    click.echo(log_data)
+
+
+@cli.command()
+@click.option("--host", required=True, help="The openQA host URL.")
+@click.option("--job-id", required=True, type=int, help="The job ID.")
+@click.option("--filename", required=True, help="The name of the log file.")
+@click.pass_context
+def get_log_filename(ctx, host, job_id, filename):
+    """Get absolute path with filename of a single log file from the cache.
+
+    The file is downloaded to the cache if not already available locally.
+    """
+    oll = openQA_log_local(host=host)
+    log_filename = oll.get_log_filename(job_id, filename)
+    click.echo(log_filename)
+
+
+if __name__ == "__main__":
+    cli(obj={})

openqa_log_local-0.0.1/src/openqa_log_local/client.py

@@ -0,0 +1,105 @@
+import logging
+from typing import Any, Optional
+
+import requests
+import requests.exceptions
+from openqa_client.client import OpenQA_Client
+from openqa_client.exceptions import RequestError
+
+"""Custom exception classes for the application."""
+
+
+class openQAClientError(Exception):
+    """Base exception for all openQAClientWrapper errors."""
+
+    pass
+
+
+class openQAClientAPIError(openQAClientError):
+    """Raised for errors during openQA API requests."""
+
+    pass
+
+
+class openQAClientConnectionError(openQAClientError):
+    """Raised for errors during connection to openQA."""
+
+    pass
+
+
+class openQAClientWrapper:
+    """A wrapper class for the openqa_client to simplify interactions."""
+
+    def __init__(
+        self,
+        hostname: str,
+        logger: logging.Logger,
+    ) -> None:
+        """Initializes the client wrapper.
+
+        It does not create an OpenQA_Client instance immediately. The client
+        is lazily initialized on first use.
+
+        Args:
+            hostname (str): The openQA host URL.
+            logger (logging.Logger): The logger instance to use.
+        """
+        self.logger = logger
+        self.hostname = hostname
+        self._client: Optional[OpenQA_Client] = None
+
+    @property
+    def client(self) -> OpenQA_Client:
+        """Lazily initializes and returns the OpenQA_Client instance.
+
+        Returns:
+            OpenQA_Client: The initialized openqa_client instance.
+        """
+        if self._client is None:
+            self.logger.info("Initializing OpenQA_Client for %s", self.hostname)
+            client = OpenQA_Client(server=self.hostname)
+            client.session.verify = False
+            self.logger.warning(
+                "SSL certificate verification disabled for client connecting to %s",
+                self.hostname,
+            )
+            self._client = client
+        return self._client
+
+    def get_job_details(self, job_id: int) -> Optional[dict[str, Any]]:
+        """Fetches the details for a specific job from the openQA API.
+
+        Args:
+            job_id (int): The ID of the job.
+
+        Raises:
+            openQAClientAPIError: For non-404 API errors.
+            openQAClientConnectionError: For network connection errors.
+
+        Returns:
+            Optional[dict[str, Any]]: A dictionary with job details, or None if the job is not found (404).
+        """
+        self.logger.info(
+            "get_job_details(job_id:%s) for hostname:%s", job_id, self.hostname
+        )
+        try:
+            response = self.client.openqa_request("GET", f"jobs/{job_id}")
+            job = response.get("job")
+            if not job:
+                raise openQAClientAPIError(
+                    f"Could not find 'job' key in API response for ID {job_id}."
+                )
+            return job
+        except RequestError as e:
+            if e.status_code == 404:
+                self.logger.warning("Job %s not found (404)", job_id)
+                return None
+            error_message = (
+                f"API Error for job {job_id}: Status {e.status_code} - {e.text}"
+            )
+            self.logger.error(error_message)
+            raise openQAClientAPIError(error_message) from e
+        except requests.exceptions.ConnectionError as e:
+            error_message = f"Connection to host '{self.hostname}' failed"
+            self.logger.error(error_message)
+            raise openQAClientConnectionError(error_message) from e

openqa_log_local-0.0.1/src/openqa_log_local/main.py

@@ -0,0 +1,129 @@
+import logging
+from typing import Any, Dict, List, Optional
+
+from .client import openQAClientWrapper
+from .cache import openQACache
+
+
+class openQA_log_local:
+    """
+    Main class for the openqa_log_local library.
+    """
+
+    def __init__(
+        self,
+        host: str,
+        cache_location: Optional[str] = ".cache",
+        max_size: Optional[int] = 1024 * 1024 * 100,  # 100 MB
+        time_to_live: Optional[int] = -1,
+        logger: Optional[logging.Logger] = None,
+    ):
+        """
+        Initializes the openQA_log_local library.
+
+        Args:
+            host (str): The openQA host URL.
+            cache_location (Optional[str]): The directory to store cached logs.
+                                        Defaults to ".cache".
+            max_size (Optional[int]): The maximum size of the cache in bytes.
+                                  Defaults to 100MB.
+            time_to_live (Optional[int]): The time in seconds after which cached
+                                        data is considered stale. -1 means
+                                        data never expires, 0 means data is
+                                        always refreshed. Defaults to -1.
+            logger (Optional[logging.Logger]): A logger instance. If None, a
+                                             new one is created.
+        """
+        if logger is None:
+            self.logger = logging.getLogger(__name__)
+        else:
+            self.logger = logger
+        self.client = openQAClientWrapper(host, self.logger)
+        if cache_location is None:
+            cl = ".cache"
+        else:
+            cl = cache_location
+        if max_size is None:
+            ms = 1024 * 1024 * 100
+        else:
+            ms = max_size
+        if time_to_live is None:
+            tl = -1
+        else:
+            tl = time_to_live
+        self.cache = openQACache(
+            cl,
+            host,
+            ms,
+            tl,
+            self.logger,
+        )
+
+    def get_details(self, job_id: int) -> Optional[Dict[str, Any]]:
+        """Get job details for a specific openQA job.
+
+        Args:
+            job_id (int): The job ID.
+
+        Returns:
+            Optional[Dict[str, Any]]: A dictionary containing job details,
+            or None if the job is not found.
+        """
+        job_details: Optional[Dict[str, Any]] = None
+        if not self.cache.is_details_cached(str(job_id)):
+            self.logger.info(f"Cache miss for job {job_id} details.")
+            job_details = self.client.get_job_details(job_id)
+            # Assuming we don't have the log files list at this point.
+            # We will update the cache when we fetch the log files.
+            if job_details:
+                self.cache.write_details(str(job_id), job_details, [])
+        else:
+            self.logger.info(f"Cache hit for job {job_id} details.")
+            job_details = self.cache.get_job_details(str(job_id))
+
+        return job_details
+
+    def get_log_list(
+        self, job_id: int, name_pattern: Optional[str] = None
+    ) -> List[str]:
+        """Get a list of log files associated to an openQA job.
+
+        This method does not download any log files.
+
+        Args:
+            job_id (int): The job ID.
+            name_pattern (Optional[str]): A regex pattern to filter log files by name.
+
+        Returns:
+            List[str]: A list of log file names.
+        """
+        return []
+
+    def get_log_data(self, job_id: int, filename: str) -> str:
+        """Get content of a single log file.
+
+        The file is downloaded to the cache if not already available locally.
+        All the log file content is returned.
+
+        Args:
+            job_id (int): The job ID.
+            filename (str): The name of the log file.
+
+        Returns:
+            str: The content of the log file.
+        """
+        return ""
+
+    def get_log_filename(self, job_id: int, filename: str) -> str:
+        """Get absolute path with filename of a single log file from the cache.
+
+        The file is downloaded to the cache if not already available locally.
+
+        Args:
+            job_id (int): The job ID.
+            filename (str): The name of the log file.
+
+        Returns:
+            str: The absolute path to the cached log file.
+        """
+        return ""