logdetective 2.12.0__tar.gz → 3.0.0__tar.gz

{logdetective-2.12.0 → logdetective-3.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: logdetective
-Version: 2.12.0
+Version: 3.0.0
 Summary: Log using LLM AI to search for build/test failures and provide ideas for fixing these.
 License: Apache-2.0
 License-File: LICENSE
@@ -24,7 +24,7 @@ Provides-Extra: server
 Provides-Extra: server-testing
 Provides-Extra: testing
 Requires-Dist: aiohttp (>=3.7.4,<4.0.0)
-Requires-Dist: aiolimiter (>=1.0.0,<2.0.0) ; extra == "server"
+Requires-Dist: aiolimiter (>=1.0.0,<2.0.0) ; extra == "server" or extra == "server-testing"
 Requires-Dist: aioresponses (>=0.7.8,<0.8.0) ; extra == "testing"
 Requires-Dist: alembic (>=1.13.3,<2.0.0) ; extra == "server" or extra == "server-testing"
 Requires-Dist: asciidoc[testing] (>=10.2.1,<11.0.0) ; extra == "testing"
@@ -36,7 +36,6 @@ Requires-Dist: flexmock (>=0.12.2,<0.13.0) ; extra == "testing"
 Requires-Dist: huggingface-hub (>=0.23.0,<1.4.0)
 Requires-Dist: koji (>=1.35.0,<2.0.0) ; extra == "server" or extra == "server-testing"
 Requires-Dist: llama-cpp-python (>0.2.56,!=0.2.86,<1.0.0)
-Requires-Dist: matplotlib (>=3.8.4,<4.0.0) ; extra == "server" or extra == "server-testing"
 Requires-Dist: numpy (>=1.26.0)
 Requires-Dist: openai (>=1.82.1,<2.0.0) ; extra == "server" or extra == "server-testing"
 Requires-Dist: pydantic (>=2.8.2,<3.0.0)
@@ -98,11 +97,13 @@ Usage
 To analyze a log file, run the script with the following command line arguments:
 - `file` (required): The path or URL of the log file to be analyzed.
 - `--model` (optional, default: "Mistral-7B-Instruct-v0.3-GGUF"): The path or Hugging space name of the language model for analysis. For models from Hugging Face, write them as `namespace/repo_name`. As we are using LLama.cpp we want this to be in the `gguf` format. If the model is already on your machine it will skip the download.
-- `--filename_suffix` (optional, default "Q4_K.gguf"): You can specify which suffix of the file to use. This option is applied when specifying model using the Hugging Face repository.
-- `--summarizer` DISABLED: LLM summarization option was removed. Argument is kept for backward compatibility only.(optional, default: "drain"): Choose between LLM and Drain template miner as the log summarizer. You can also provide the path to an existing language model file instead of using a URL.
-- `--n_lines` DISABLED: LLM summarization option was removed. Argument is kept for backward compatibility only. (optional, default: 8): The number of lines per chunk for LLM analysis. This only makes sense when you are summarizing with LLM.
-- `--n_clusters` (optional, default 8): Number of clusters for Drain to organize log chunks into. This only makes sense when you are summarizing with Drain.
-- `--skip_snippets` Path to patterns for skipping snippets (in YAML).
+- `--filename-suffix` (optional, default "Q4_K.gguf"): You can specify which suffix of the file to use. This option is applied when specifying model using the Hugging Face repository.
+- `--n-clusters` (optional, default 8): Number of clusters for Drain to organize log chunks into. This only makes sense when you are summarizing with Drain.
+- `--skip-snippets` Path to patterns for skipping snippets (in YAML).
+- `--prompts PROMPTS` Path to prompt configuration file.
+- `--temperature` Temperature for inference.
+- `--skip-snippets` Path to patterns for skipping snippets.
+- `--csgrep` Use csgrep to process the log.
 
 Example usage:
 
@@ -112,9 +113,9 @@ Or if the log file is stored locally:
 
     logdetective ./data/logs.txt
 
-Examples of using different models. Note the use of `--filename_suffix` (or `-F`) option, useful for models that were quantized:
+Examples of using different models. Note the use of `--filename-suffix` (or `-F`) option, useful for models that were quantized:
 
-    logdetective https://example.com/logs.txt --model QuantFactory/Meta-Llama-3-8B-Instruct-GGUF --filename_suffix Q5_K_S.gguf
+    logdetective https://example.com/logs.txt --model QuantFactory/Meta-Llama-3-8B-Instruct-GGUF --filename-suffix Q5_K_S.gguf
     logdetective https://kojipkgs.fedoraproject.org//work/tasks/3367/131313367/build.log --model 'fedora-copr/granite-3.2-8b-instruct-GGUF' -F Q4_K_M.gguf
 
 Example of altered prompts:
@@ -126,7 +127,7 @@ Example of altered prompts:
 
 Note that streaming with some models (notably Meta-Llama-3) is broken and can be worked around by `no-stream` option:
 
-    logdetective https://example.com/logs.txt --model QuantFactory/Meta-Llama-3-8B-Instruct-GGUF --filename_suffix Q5_K_M.gguf --no-stream
+    logdetective https://example.com/logs.txt --model QuantFactory/Meta-Llama-3-8B-Instruct-GGUF --filename-suffix Q5_K_M.gguf --no-stream
 
 Choice of LLM
 -------------

{logdetective-2.12.0 → logdetective-3.0.0}/README.md

@@ -43,11 +43,13 @@ Usage
 To analyze a log file, run the script with the following command line arguments:
 - `file` (required): The path or URL of the log file to be analyzed.
 - `--model` (optional, default: "Mistral-7B-Instruct-v0.3-GGUF"): The path or Hugging space name of the language model for analysis. For models from Hugging Face, write them as `namespace/repo_name`. As we are using LLama.cpp we want this to be in the `gguf` format. If the model is already on your machine it will skip the download.
-- `--filename_suffix` (optional, default "Q4_K.gguf"): You can specify which suffix of the file to use. This option is applied when specifying model using the Hugging Face repository.
-- `--summarizer` DISABLED: LLM summarization option was removed. Argument is kept for backward compatibility only.(optional, default: "drain"): Choose between LLM and Drain template miner as the log summarizer. You can also provide the path to an existing language model file instead of using a URL.
-- `--n_lines` DISABLED: LLM summarization option was removed. Argument is kept for backward compatibility only. (optional, default: 8): The number of lines per chunk for LLM analysis. This only makes sense when you are summarizing with LLM.
-- `--n_clusters` (optional, default 8): Number of clusters for Drain to organize log chunks into. This only makes sense when you are summarizing with Drain.
-- `--skip_snippets` Path to patterns for skipping snippets (in YAML).
+- `--filename-suffix` (optional, default "Q4_K.gguf"): You can specify which suffix of the file to use. This option is applied when specifying model using the Hugging Face repository.
+- `--n-clusters` (optional, default 8): Number of clusters for Drain to organize log chunks into. This only makes sense when you are summarizing with Drain.
+- `--skip-snippets` Path to patterns for skipping snippets (in YAML).
+- `--prompts PROMPTS` Path to prompt configuration file.
+- `--temperature` Temperature for inference.
+- `--skip-snippets` Path to patterns for skipping snippets.
+- `--csgrep` Use csgrep to process the log.
 
 Example usage:
 
@@ -57,9 +59,9 @@ Or if the log file is stored locally:
 
     logdetective ./data/logs.txt
 
-Examples of using different models. Note the use of `--filename_suffix` (or `-F`) option, useful for models that were quantized:
+Examples of using different models. Note the use of `--filename-suffix` (or `-F`) option, useful for models that were quantized:
 
-    logdetective https://example.com/logs.txt --model QuantFactory/Meta-Llama-3-8B-Instruct-GGUF --filename_suffix Q5_K_S.gguf
+    logdetective https://example.com/logs.txt --model QuantFactory/Meta-Llama-3-8B-Instruct-GGUF --filename-suffix Q5_K_S.gguf
     logdetective https://kojipkgs.fedoraproject.org//work/tasks/3367/131313367/build.log --model 'fedora-copr/granite-3.2-8b-instruct-GGUF' -F Q4_K_M.gguf
 
 Example of altered prompts:
@@ -71,7 +73,7 @@ Example of altered prompts:
 
 Note that streaming with some models (notably Meta-Llama-3) is broken and can be worked around by `no-stream` option:
 
-    logdetective https://example.com/logs.txt --model QuantFactory/Meta-Llama-3-8B-Instruct-GGUF --filename_suffix Q5_K_M.gguf --no-stream
+    logdetective https://example.com/logs.txt --model QuantFactory/Meta-Llama-3-8B-Instruct-GGUF --filename-suffix Q5_K_M.gguf --no-stream
 
 Choice of LLM
 -------------

{logdetective-2.12.0 → logdetective-3.0.0}/logdetective/logdetective.py

@@ -41,31 +41,15 @@ def setup_args():
     )
     parser.add_argument(
         "-F",
-        "--filename_suffix",
+        "--filename-suffix",
         help="Suffix of the model file name to be retrieved from Hugging Face.\
                             Makes sense only if the model is specified with Hugging Face name.",
         default="Q4_K.gguf",
     )
     parser.add_argument("-n", "--no-stream", action="store_true")
-    parser.add_argument(
-        "-S",
-        "--summarizer",
-        type=str,
-        default="drain",
-        help="DISABLED: LLM summarization option was removed. \
-                Argument is kept for backward compatibility only.",
-    )
-    parser.add_argument(
-        "-N",
-        "--n_lines",
-        type=int,
-        default=None,
-        help="DISABLED: LLM summarization option was removed. \
-                Argument is kept for backward compatibility only.",
-    )
     parser.add_argument(
         "-C",
-        "--n_clusters",
+        "--n-clusters",
         type=int,
         default=8,
         help="Number of clusters for Drain to organize log chunks into.\
@@ -86,7 +70,7 @@ def setup_args():
         help="Temperature for inference.",
     )
     parser.add_argument(
-        "--skip_snippets",
+        "--skip-snippets",
         type=str,
         default=f"{os.path.dirname(__file__)}/skip_snippets.yml",
         help="Path to patterns for skipping snippets.",
@@ -105,10 +89,6 @@ async def run():  # pylint: disable=too-many-statements,too-many-locals,too-many
         sys.stderr.write("Error: --quiet and --verbose is mutually exclusive.\n")
         sys.exit(2)
 
-    # Emit warning about use of discontinued args
-    if args.n_lines or args.summarizer != "drain":
-        LOG.warning("LLM based summarization was removed. Drain will be used instead.")
-
     # Logging facility setup
     log_level = logging.INFO
     if args.verbose >= 1:

{logdetective-2.12.0 → logdetective-3.0.0}/logdetective/server/database/base.py

@@ -22,7 +22,7 @@ sqlalchemy_echo = getenv("SQLALCHEMY_ECHO", "False").lower() in (
     "y",
     "1",
 )
-engine = create_async_engine(get_pg_url(), echo=sqlalchemy_echo)
+engine = create_async_engine(get_pg_url(), echo=sqlalchemy_echo, pool_pre_ping=True)
 SessionFactory = async_sessionmaker(autoflush=True, bind=engine)  # pylint: disable=invalid-name
 
 

{logdetective-2.12.0 → logdetective-3.0.0}/logdetective/server/database/models/metrics.py

@@ -314,10 +314,13 @@ class AnalyzeRequestMetrics(Base):
                         "time_range"
                     ),
                     (
-                        func.avg(
-                            func.extract(  # pylint: disable=not-callable
-                                "epoch", cls.response_sent_at - cls.request_received_at
-                            )
+                        func.coalesce(
+                            func.avg(
+                                func.extract(  # pylint: disable=not-callable
+                                    "epoch", cls.response_sent_at - cls.request_received_at
+                                )
+                            ),
+                            0
                         )
                     ).label("average_response_seconds"),
                 )

logdetective-3.0.0/logdetective/server/metric.py

@@ -0,0 +1,320 @@
+import inspect
+from collections import defaultdict
+import datetime
+from typing import Optional, Union, Dict
+from functools import wraps
+
+import aiohttp
+import numpy
+from starlette.responses import StreamingResponse
+
+from logdetective.remote_log import RemoteLog
+from logdetective.server.config import LOG
+from logdetective.server.compressors import LLMResponseCompressor, RemoteLogCompressor
+from logdetective.server.models import (
+    TimePeriod,
+    MetricTimeSeries,
+    StagedResponse,
+    Response,
+    Explanation,
+)
+from logdetective.server.database.models import EndpointType, AnalyzeRequestMetrics, Reactions
+from logdetective.server.exceptions import LogDetectiveMetricsError
+
+
+async def add_new_metrics(
+    api_name: EndpointType,
+    url: Optional[str] = None,
+    http_session: Optional[aiohttp.ClientSession] = None,
+    received_at: Optional[datetime.datetime] = None,
+    compressed_log_content: Optional[bytes] = None,
+) -> int:
+    """Add a new database entry for a received request.
+
+    This will store the time when this function is called,
+    the endpoint from where the request was received,
+    and the log (in a zip format) for which analysis is requested.
+    """
+    if not compressed_log_content:
+        if not (url and http_session):
+            raise LogDetectiveMetricsError(
+                f"""Remote log can not be retrieved without URL and http session.
+                URL: {url}, http session:{http_session}""")
+        remote_log = RemoteLog(url, http_session)
+        compressed_log_content = await RemoteLogCompressor(remote_log).zip_content()
+
+    return await AnalyzeRequestMetrics.create(
+        endpoint=EndpointType(api_name),
+        compressed_log=compressed_log_content,
+        request_received_at=received_at
+        if received_at
+        else datetime.datetime.now(datetime.timezone.utc),
+    )
+
+
+async def update_metrics(
+    metrics_id: int,
+    response: Union[Response, StagedResponse, StreamingResponse],
+    sent_at: Optional[datetime.datetime] = None,
+) -> None:
+    """Update a database metric entry for a received request,
+    filling data for the given response.
+
+    This will add to the database entry the time when the response was sent,
+    the length of the created response and the certainty for it.
+    """
+    try:
+        compressed_response = LLMResponseCompressor(response).zip_response()
+    except AttributeError as e:
+        compressed_response = None
+        LOG.warning(
+            "Given response can not be serialized "
+            "and saved in db (probably a StreamingResponse): %s.",
+            e,
+        )
+
+    response_sent_at = (
+        sent_at if sent_at else datetime.datetime.now(datetime.timezone.utc)
+    )
+    response_length = None
+    if hasattr(response, "explanation") and isinstance(
+        response.explanation, Explanation
+    ):
+        response_length = len(response.explanation.text)
+    response_certainty = (
+        response.response_certainty if hasattr(response, "response_certainty") else None
+    )
+    await AnalyzeRequestMetrics.update(
+        id_=metrics_id,
+        response_sent_at=response_sent_at,
+        response_length=response_length,
+        response_certainty=response_certainty,
+        compressed_response=compressed_response,
+    )
+
+
+def track_request(name=None):
+    """
+    Decorator to track requests/responses metrics
+
+    On entering the decorated function, it registers the time for the request
+    and saves the passed log content.
+    On exiting the decorated function, it registers the time for the response
+    and saves the generated response.
+
+    Use it to decorate server endpoints that generate a llm response
+    as in the following example:
+
+    >>> @app.post("/analyze", response_model=Response)
+    >>> @track_request()
+    >>> async def analyze_log(build_log)
+    >>>     pass
+
+    Warning: the decorators' order is important!
+    The function returned by the *track_request* decorator is the
+    server API function we want to be called by FastAPI.
+    """
+
+    def decorator(f):
+        @wraps(f)
+        async def async_decorated_function(*args, **kwargs):
+            log_url = kwargs["build_log"].url
+            metrics_id = await add_new_metrics(
+                api_name=EndpointType(name if name else f.__name__),
+                url=log_url, http_session=kwargs["http_session"]
+            )
+            response = await f(*args, **kwargs)
+            await update_metrics(metrics_id, response)
+            return response
+
+        if inspect.iscoroutinefunction(f):
+            return async_decorated_function
+        raise NotImplementedError("An async coroutine is needed")
+
+    return decorator
+
+
+# TODO: Refactor aggregation to use database operations, instead of timestamp formatting  # pylint: disable=fixme
+class TimeDefinition:
+    """Define time format details, given a time period."""
+
+    def __init__(self, time_period: TimePeriod):
+        self.time_period = time_period
+        self.days_diff = time_period.get_time_period().days
+        if self.time_period.hours:
+            self._time_format = "%Y-%m-%d %H"
+            self._time_delta = datetime.timedelta(hours=1)
+        elif self.time_period.days:
+            self._time_format = "%Y-%m-%d"
+            self._time_delta = datetime.timedelta(days=1)
+        elif self.time_period.weeks:
+            self._time_format = "%Y-%m-%d"
+            self._time_delta = datetime.timedelta(weeks=1)
+
+    @property
+    def time_format(self):
+        # pylint: disable=missing-function-docstring
+        return self._time_format
+
+    @property
+    def time_delta(self):
+        # pylint: disable=missing-function-docstring
+        return self._time_delta
+
+
+def create_time_series_arrays(
+    values_dict: dict[datetime.datetime, int],
+) -> tuple[list, list]:
+    """Create time series arrays from a dictionary of values.
+
+    This function generates two aligned lists:
+    1. An array of timestamps from start_time to end_time
+    2. A corresponding array of values for each timestamp
+
+    Args:
+        values_dict: Dictionary mapping timestamps to their respective values
+    Returns:
+        A tuple containing:
+            - list: Array of timestamps
+            - list: Array of corresponding values
+    """
+
+    timestamps = []
+    values = []
+
+    for timestamp, count in values_dict.items():
+        timestamps.append(timestamp)
+        values.append(count)
+
+    return timestamps, numpy.nan_to_num(values).tolist()
+
+
+async def requests_per_time(
+    period_of_time: TimePeriod,
+    endpoint: EndpointType = EndpointType.ANALYZE,
+    end_time: Optional[datetime.datetime] = None,
+) -> MetricTimeSeries:
+    """
+    Get request counts over a specified time period.
+
+    The time intervals are determined by the provided TimePeriod object, which defines
+    the granularity.
+
+    Args:
+        period_of_time: A TimePeriod object that defines the time period and interval
+                        for the analysis (e.g., hourly, daily, weekly)
+        endpoint: One of the API endpoints
+        end_time: The end time for the analysis period. If None, defaults to the current
+                  UTC time
+
+    Returns:
+        A dictionary with timestamps and associated number of requests
+    """
+    end_time = end_time or datetime.datetime.now(datetime.timezone.utc)
+    start_time = period_of_time.get_period_start_time(end_time)
+    time_def = TimeDefinition(period_of_time)
+    requests_counts = await AnalyzeRequestMetrics.get_requests_in_period(
+        start_time, end_time, time_def.time_format, endpoint
+    )
+    timestamps, counts = create_time_series_arrays(requests_counts)
+
+    return MetricTimeSeries(metric="requests", timestamps=timestamps, values=counts)
+
+
+async def average_time_per_responses(
+    period_of_time: TimePeriod,
+    endpoint: EndpointType = EndpointType.ANALYZE,
+    end_time: Optional[datetime.datetime] = None,
+) -> MetricTimeSeries:
+    """
+    Get average response time and length over a specified time period.
+
+    The time intervals are determined by the provided TimePeriod object, which defines
+    the granularity.
+
+    Args:
+        period_of_time: A TimePeriod object that defines the time period and interval
+                        for the analysis (e.g., hourly, daily, weekly)
+        endpoint: One of the API endpoints
+        end_time: The end time for the analysis period. If None, defaults to the current
+                  UTC time
+
+    Returns:
+        A dictionary of timestamps and average response times
+    """
+    end_time = end_time or datetime.datetime.now(datetime.timezone.utc)
+    start_time = period_of_time.get_period_start_time(end_time)
+    time_def = TimeDefinition(period_of_time)
+    responses_average_time = (
+        await AnalyzeRequestMetrics.get_responses_average_time_in_period(
+            start_time, end_time, time_def.time_format, endpoint
+        )
+    )
+    timestamps, average_time = create_time_series_arrays(
+        responses_average_time,
+    )
+
+    return MetricTimeSeries(metric="avg_response_time", timestamps=timestamps, values=average_time)
+
+
+async def _collect_emoji_data(
+    start_time: datetime.datetime, time_def: TimeDefinition
+) -> Dict[str, Dict[str, list]]:
+    """Collect and organize emoji feedback data
+
+    For each reaction type, a dictionary is created with time stamps
+    as keys, and aggregate counts as values.
+    """
+    reactions = await Reactions.get_since(start_time)
+    reaction_values: defaultdict[str, Dict] = defaultdict(lambda: defaultdict(int))
+
+    for comment_timestamp, reaction in reactions:
+        formatted_timestamp = comment_timestamp.strptime(
+            comment_timestamp.strftime(time_def.time_format), time_def.time_format
+        )
+
+        reaction_values[reaction.reaction_type][formatted_timestamp] += reaction.count
+
+    reaction_time_series = {
+        reaction_type: {
+            "timestamps": reaction_data.keys(),
+            "values": reaction_data.values(),
+        }
+        for reaction_type, reaction_data in reaction_values.items()
+    }
+
+    return reaction_time_series
+
+
+async def emojis_per_time(
+    period_of_time: TimePeriod,
+    end_time: Optional[datetime.datetime] = None,
+) -> list[MetricTimeSeries]:
+    """
+    Retrieve data of emoji feedback over time.
+
+    The time intervals are determined by the provided TimePeriod object, which defines
+    the granularity.
+
+    Args:
+        period_of_time: A TimePeriod object that defines the time period and interval
+                        for the analysis (e.g., hourly, daily, weekly)
+        end_time: The end time for the analysis period. If None, defaults to the current
+                  UTC time
+
+    Returns:
+        A list of `MetricTimeSeries` objects
+    """
+    time_def = TimeDefinition(period_of_time)
+    end_time = end_time or datetime.datetime.now(datetime.timezone.utc)
+    start_time = period_of_time.get_period_start_time(end_time)
+    reactions_values_dict = await _collect_emoji_data(start_time, time_def)
+
+    reaction_values: list[MetricTimeSeries] = []
+    for reaction, time_series in reactions_values_dict.items():
+        reaction_values.append(
+            MetricTimeSeries(
+                metric=f"emoji_{reaction}",
+                timestamps=time_series["timestamps"],
+                values=time_series["values"]))
+    return reaction_values

{logdetective-2.12.0 → logdetective-3.0.0}/logdetective/server/models.py

@@ -401,3 +401,15 @@ class TimePeriod(BaseModel):
         if time.tzinfo is None:
             time = time.replace(tzinfo=datetime.timezone.utc)
         return time - self.get_time_period()
+
+
+class MetricTimeSeries(BaseModel):
+    """Recorded values of given metric"""
+    metric: str
+    timestamps: List[datetime.datetime]
+    values: List[float]
+
+
+class MetricResponse(BaseModel):
+    """Requested metrics"""
+    time_series: List[MetricTimeSeries]