logdetective 1.9.0__tar.gz → 2.0.1__tar.gz

{logdetective-1.9.0 → logdetective-2.0.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: logdetective
-Version: 1.9.0
+Version: 2.0.1
 Summary: Log using LLM AI to search for build/test failures and provide ideas for fixing these.
 License: Apache-2.0
 Author: Jiri Podivin
@@ -124,6 +124,7 @@ Note that streaming with some models (notably Meta-Llama-3 is broken) is broken
 
 Real Example
 ------------
+
 Let's have a look at a real world example. Log Detective can work with any logs though we optimize it for RPM build logs.
 
 We're going to analyze a failed build of a python-based library that happened in Fedora Koji buildsystem:
@@ -187,7 +188,7 @@ It looks like a wall of text. Similar to any log. The main difference is that he
 
 
 Contributing
-------------
+============
 
 Contributions are welcome! Please submit a pull request if you have any improvements or new features to add. Make sure your changes pass all existing tests before submitting.
 For bigger code changes, please consult us first by creating an issue.
@@ -304,7 +305,7 @@ podman-compose up server
 - Run Visual Stdio Code debug configuration named *Python Debug: Remote Attach*
 
 Server
-------
+======
 
 FastApi based server is implemented in `logdetective/server.py`. In order to run it in a development mode,
 simply start llama-cpp-python server with your chosen model as described in llama-cpp-python [docs](https://llama-cpp-python.readthedocs.io/en/latest/server/#running-the-server).
@@ -335,6 +336,30 @@ Model can be downloaded from [our Hugging Space](https://huggingface.co/fedora-c
 $ curl -L -o models/mistral-7b-instruct-v0.3.Q4_K.gguf https://huggingface.co/fedora-copr/Mistral-7B-Instruct-v0.3-GGUF/resolve/main/ggml-model-Q4_K.gguf
 ```
 
+Filtering snippet analysis by relevance
+---------------------------------------
+
+When using `/analyze/staged` API, it is possible to enable filtering analyzed snippets by their estimated relavance, submitting only those with highest meansure of relevance for final analysis.
+
+**Note**: This feautre requires LLM provider with support for JSON structured output. Smaller models, even though techically capable of providing structured output, may not be able to appropriatelly estimate snippet relevance.
+
+Filtering is disabled by default and must be enabled by setting value of `top_k_snippets` field in `general` section of server configuration. Value indicates number of snippets with highest estimated relavance that will be submitted for final analysis.
+
+Example:
+
+```
+general:
+  devmode: False
+  packages:
+    - .*
+  excluded_packages:
+    - ^redhat-internal-.*
+  top_k_snippets: 10
+```
+
+If all snippets are rated the same, the filtering is skipped and warning raised in logs.
+Values higher than total number of snippets, as set by `max_clusters` in the `extrator` section of config, also result in filtering being skipped.
+
 Generate a new database revision with alembic
 ---------------------------------------------
 

{logdetective-1.9.0 → logdetective-2.0.1}/README.md

@@ -79,6 +79,7 @@ Note that streaming with some models (notably Meta-Llama-3 is broken) is broken
 
 Real Example
 ------------
+
 Let's have a look at a real world example. Log Detective can work with any logs though we optimize it for RPM build logs.
 
 We're going to analyze a failed build of a python-based library that happened in Fedora Koji buildsystem:
@@ -142,7 +143,7 @@ It looks like a wall of text. Similar to any log. The main difference is that he
 
 
 Contributing
-------------
+============
 
 Contributions are welcome! Please submit a pull request if you have any improvements or new features to add. Make sure your changes pass all existing tests before submitting.
 For bigger code changes, please consult us first by creating an issue.
@@ -259,7 +260,7 @@ podman-compose up server
 - Run Visual Stdio Code debug configuration named *Python Debug: Remote Attach*
 
 Server
-------
+======
 
 FastApi based server is implemented in `logdetective/server.py`. In order to run it in a development mode,
 simply start llama-cpp-python server with your chosen model as described in llama-cpp-python [docs](https://llama-cpp-python.readthedocs.io/en/latest/server/#running-the-server).
@@ -290,6 +291,30 @@ Model can be downloaded from [our Hugging Space](https://huggingface.co/fedora-c
 $ curl -L -o models/mistral-7b-instruct-v0.3.Q4_K.gguf https://huggingface.co/fedora-copr/Mistral-7B-Instruct-v0.3-GGUF/resolve/main/ggml-model-Q4_K.gguf
 ```
 
+Filtering snippet analysis by relevance
+---------------------------------------
+
+When using `/analyze/staged` API, it is possible to enable filtering analyzed snippets by their estimated relavance, submitting only those with highest meansure of relevance for final analysis.
+
+**Note**: This feautre requires LLM provider with support for JSON structured output. Smaller models, even though techically capable of providing structured output, may not be able to appropriatelly estimate snippet relevance.
+
+Filtering is disabled by default and must be enabled by setting value of `top_k_snippets` field in `general` section of server configuration. Value indicates number of snippets with highest estimated relavance that will be submitted for final analysis.
+
+Example:
+
+```
+general:
+  devmode: False
+  packages:
+    - .*
+  excluded_packages:
+    - ^redhat-internal-.*
+  top_k_snippets: 10
+```
+
+If all snippets are rated the same, the filtering is skipped and warning raised in logs.
+Values higher than total number of snippets, as set by `max_clusters` in the `extrator` section of config, also result in filtering being skipped.
+
 Generate a new database revision with alembic
 ---------------------------------------------
 

{logdetective-1.9.0 → logdetective-2.0.1}/logdetective/extractors.py

@@ -20,7 +20,8 @@ class DrainExtractor:
         context: bool = False,
         max_clusters=8,
         skip_snippets: SkipSnippets = SkipSnippets({}),
-    ):
+        max_snippet_len: int = 2000
+    ):  # pylint: disable=R0913,R0917
         config = TemplateMinerConfig()
         config.load(f"{os.path.dirname(__file__)}/drain3.ini")
         config.profiling_enabled = verbose
@@ -29,11 +30,12 @@ class DrainExtractor:
         self.verbose = verbose
         self.context = context
         self.skip_snippets = skip_snippets
+        self.max_snippet_len = max_snippet_len
 
     def __call__(self, log: str) -> list[Tuple[int, str]]:
         out = []
         # Create chunks
-        chunks = list(get_chunks(log))
+        chunks = list(get_chunks(log, self.max_snippet_len))
         # Keep only chunks that don't match any of the excluded patterns
         chunks = [
             (_, chunk)

{logdetective-1.9.0 → logdetective-2.0.1}/logdetective/prompts.yml

@@ -22,8 +22,8 @@ prompt_template: |
   Analysis:
 
 snippet_prompt_template: |
-  Analyse following RPM build log snippet. Describe contents accurately, without speculation or suggestions for resolution.
-
+  Analyse following RPM build log snippet. Describe contents accurately, without speculation or suggestions for resolution
+  and provide estimate of snippet relevance.
   Your analysis must be as concise as possible, while keeping relevant information intact.
 
   Snippet:

{logdetective-1.9.0 → logdetective-2.0.1}/logdetective/server/gitlab.py

@@ -434,7 +434,7 @@ async def generate_mr_comment(
     content = tpl.render(
         package=job.project_name,
         explanation=response.explanation.text,
-        certainty=f"{response.response_certainty:.2f}",
+        certainty=response.response_certainty,
         emoji_face=emoji_face,
         snippets=response.snippets,
         log_url=log_url,

{logdetective-1.9.0 → logdetective-2.0.1}/logdetective/server/koji.py

@@ -4,7 +4,6 @@ from typing import Any, Callable, Optional
 
 import backoff
 import koji
-from logdetective.server.config import LOG
 from logdetective.server.exceptions import (
     KojiInvalidTaskID,
     LogDetectiveConnectionError,
@@ -12,23 +11,16 @@ from logdetective.server.exceptions import (
     LogsTooLargeError,
     UnknownTaskType,
 )
-
+from logdetective.server.utils import connection_error_giveup
 
 FAILURE_LOG_REGEX = re.compile(r"(\w*\.log)")
 
 
-def connection_error_giveup(details: backoff._typing.Details) -> None:
-    """
-    Too many connection errors, give up.
-    """
-    LOG.error("Too many connection errors, giving up. %s", details["exception"])
-    raise LogDetectiveConnectionError() from details["exception"]
-
-
 @backoff.on_exception(
     backoff.expo,
     koji.GenericError,
     max_time=60,
+    on_giveup=connection_error_giveup,
 )
 async def call_koji(func: Callable, *args, **kwargs) -> Any:
     """

logdetective-2.0.1/logdetective/server/llm.py

@@ -0,0 +1,300 @@
+import os
+import asyncio
+import random
+from typing import List, Tuple, Dict
+
+import backoff
+from fastapi import HTTPException
+from pydantic import ValidationError
+
+import aiohttp
+from openai import AsyncStream
+from openai.types.chat import ChatCompletionChunk
+
+from logdetective.utils import (
+    compute_certainty,
+    prompt_to_messages,
+    format_snippets,
+)
+from logdetective.server.config import (
+    LOG,
+    SERVER_CONFIG,
+    PROMPT_CONFIG,
+    CLIENT,
+)
+from logdetective.server.models import (
+    AnalyzedSnippet,
+    InferenceConfig,
+    Explanation,
+    StagedResponse,
+    SnippetAnalysis,
+    RatedSnippetAnalysis,
+    Response,
+)
+from logdetective.server.utils import (
+    format_analyzed_snippets,
+    mine_logs,
+    should_we_giveup,
+    we_give_up,
+    filter_snippets,
+)
+
+
+LLM_CPP_SERVER_TIMEOUT = os.environ.get("LLAMA_CPP_SERVER_TIMEOUT", 600)
+
+
+@backoff.on_exception(
+    lambda: backoff.constant([10, 30, 120]),
+    aiohttp.ClientResponseError,
+    max_tries=4,  # 4 tries and 3 retries
+    jitter=lambda wait_gen_value: random.uniform(wait_gen_value, wait_gen_value + 30),
+    giveup=should_we_giveup,
+    raise_on_giveup=False,
+    on_giveup=we_give_up,
+)
+async def call_llm(
+    messages: List[Dict[str, str]],
+    inference_cfg: InferenceConfig,
+    stream: bool = False,
+    structured_output: dict | None = None,
+) -> Explanation:
+    """Submit prompt to LLM.
+    inference_cfg: The configuration section from the config.json representing
+    the relevant inference server for this request.
+    """
+    LOG.info("Analyzing the text")
+
+    LOG.info("Submitting to /v1/chat/completions endpoint")
+
+    kwargs = {}
+
+    # OpenAI API does not guarantee that the behavior for parameter set to `None`
+    # and parameter not given at all is the same.
+    # We build a dictionary of parameters based on the configuration.
+    if inference_cfg.log_probs:
+        LOG.info("Requesting log probabilities from LLM")
+        kwargs["logprobs"] = inference_cfg.log_probs
+    if structured_output:
+        LOG.info("Requesting structured output from LLM")
+        response_format = {
+            "type": "json_schema",
+            "json_schema": {
+                "name": "rated-snippet-analysis",
+                "schema": structured_output,
+            },
+        }
+        kwargs["response_format"] = response_format
+
+    async with inference_cfg.get_limiter():
+        response = await CLIENT.chat.completions.create(
+            messages=messages,
+            max_tokens=inference_cfg.max_tokens,
+            stream=stream,
+            model=inference_cfg.model,
+            temperature=inference_cfg.temperature,
+            **kwargs,
+        )
+
+    if not response.choices[0].message.content:
+        LOG.error("No response content recieved from %s", inference_cfg.url)
+        raise RuntimeError()
+
+    message_content = response.choices[0].message.content
+
+    if response.choices[0].logprobs and response.choices[0].logprobs.content:
+        logprobs = [e.to_dict() for e in response.choices[0].logprobs.content]
+    else:
+        logprobs = None
+
+    return Explanation(
+        text=message_content,
+        logprobs=logprobs,
+    )
+
+
+@backoff.on_exception(
+    lambda: backoff.constant([10, 30, 120]),
+    aiohttp.ClientResponseError,
+    max_tries=4,  # 4 tries and 3 retries
+    jitter=lambda wait_gen_value: random.uniform(wait_gen_value, wait_gen_value + 30),
+    giveup=should_we_giveup,
+    raise_on_giveup=False,
+    on_giveup=we_give_up,
+)
+async def call_llm_stream(
+    messages: List[Dict[str, str]],
+    inference_cfg: InferenceConfig,
+    stream: bool = False,
+) -> AsyncStream[ChatCompletionChunk]:
+    """Submit prompt to LLM and recieve stream of tokens as a result.
+    inference_cfg: The configuration section from the config.json representing
+    the relevant inference server for this request.
+    """
+    LOG.info("Analyzing the text")
+
+    LOG.info("Submitting to /v1/chat/completions endpoint")
+
+    async with inference_cfg.get_limiter():
+        response = await CLIENT.chat.completions.create(
+            messages=messages,
+            max_tokens=inference_cfg.max_tokens,
+            logprobs=inference_cfg.log_probs,
+            stream=stream,
+            model=inference_cfg.model,
+            temperature=inference_cfg.temperature,
+        )
+
+    return response
+
+
+async def analyze_snippets(
+    log_summary: List[Tuple[int, str]], structured_output: dict | None = None
+) -> List[SnippetAnalysis | RatedSnippetAnalysis]:
+    """Submit log file snippets to the LLM and gather results"""
+    # Process snippets asynchronously
+    awaitables = [
+        call_llm(
+            prompt_to_messages(
+                PROMPT_CONFIG.snippet_prompt_template.format(s),
+                PROMPT_CONFIG.snippet_system_prompt,
+                SERVER_CONFIG.inference.system_role,
+                SERVER_CONFIG.inference.user_role,
+            ),
+            inference_cfg=SERVER_CONFIG.snippet_inference,
+            structured_output=structured_output,
+        )
+        for s in log_summary
+    ]
+    gathered_responses = await asyncio.gather(*awaitables)
+    analyzed_snippets = []
+
+    for response in gathered_responses:
+        if structured_output:
+            try:
+                snippet = RatedSnippetAnalysis.model_validate_json(response.text)
+            except ValidationError as ex:
+                LOG.error("Invalid data structure returned `%s`", response.text)
+                raise ex
+        else:
+            snippet = SnippetAnalysis(text=response.text)
+        analyzed_snippets.append(snippet)
+
+    return analyzed_snippets
+
+
+async def perfrom_analysis(log_text: str) -> Response:
+    """Sumbit log file snippets in aggregate to LLM and retrieve results"""
+    log_summary = mine_logs(log_text)
+    log_summary = format_snippets(log_summary)
+    messages = prompt_to_messages(
+        PROMPT_CONFIG.prompt_template.format(log_summary),
+        PROMPT_CONFIG.default_system_prompt,
+        SERVER_CONFIG.inference.system_role,
+        SERVER_CONFIG.inference.user_role,
+    )
+    response = await call_llm(
+        messages,
+        inference_cfg=SERVER_CONFIG.inference,
+    )
+    certainty = 0
+
+    if response.logprobs is not None:
+        try:
+            certainty = compute_certainty(response.logprobs)
+        except ValueError as ex:
+            LOG.error("Error encountered while computing certainty: %s", ex)
+            raise HTTPException(
+                status_code=400,
+                detail=f"Couldn't compute certainty with data:\n{response.logprobs}",
+            ) from ex
+
+    return Response(explanation=response, response_certainty=certainty)
+
+
+async def perform_analyis_stream(log_text: str) -> AsyncStream:
+    """Submit log file snippets in aggregate and return a stream of tokens"""
+    log_summary = mine_logs(log_text)
+    log_summary = format_snippets(log_summary)
+    messages = prompt_to_messages(
+        PROMPT_CONFIG.prompt_template.format(log_summary),
+        PROMPT_CONFIG.default_system_prompt,
+        SERVER_CONFIG.inference.system_role,
+        SERVER_CONFIG.inference.user_role,
+    )
+
+    stream = call_llm_stream(
+        messages,
+        inference_cfg=SERVER_CONFIG.inference,
+    )
+
+    # we need to figure out a better response here, this is how it looks rn:
+    # b'data: {"choices":[{"finish_reason":"stop","index":0,"delta":{}}],
+    #   "created":1744818071,"id":"chatcmpl-c9geTxNcQO7M9wR...
+    return stream
+
+
+async def perform_staged_analysis(log_text: str) -> StagedResponse:
+    """Submit the log file snippets to the LLM and retrieve their results"""
+    log_summary = mine_logs(log_text)
+
+    if SERVER_CONFIG.general.top_k_snippets:
+        rated_snippets = await analyze_snippets(
+            log_summary=log_summary,
+            structured_output=RatedSnippetAnalysis.model_json_schema(),
+        )
+
+        # Extract original text and line number from `log_summary`
+        processed_snippets = [
+            AnalyzedSnippet(line_number=e[0][0], text=e[0][1], explanation=e[1])
+            for e in zip(log_summary, rated_snippets)
+        ]
+        processed_snippets = filter_snippets(
+            processed_snippets=processed_snippets,
+            top_k=SERVER_CONFIG.general.top_k_snippets,
+        )
+        LOG.info(
+            "Keeping %d of original %d snippets",
+            len(processed_snippets),
+            len(rated_snippets),
+        )
+    else:
+        processed_snippets = await analyze_snippets(log_summary=log_summary)
+
+        # Extract original text and line number from `log_summary`
+        processed_snippets = [
+            AnalyzedSnippet(line_number=e[0][0], text=e[0][1], explanation=e[1])
+            for e in zip(log_summary, processed_snippets)
+        ]
+
+    final_prompt = PROMPT_CONFIG.prompt_template_staged.format(
+        format_analyzed_snippets(processed_snippets)
+    )
+    messages = prompt_to_messages(
+        final_prompt,
+        PROMPT_CONFIG.staged_system_prompt,
+        SERVER_CONFIG.inference.system_role,
+        SERVER_CONFIG.inference.user_role,
+    )
+    final_analysis = await call_llm(
+        messages,
+        inference_cfg=SERVER_CONFIG.inference,
+    )
+
+    certainty = 0
+
+    if final_analysis.logprobs:
+        try:
+            certainty = compute_certainty(final_analysis.logprobs)
+        except ValueError as ex:
+            LOG.error("Error encountered while computing certainty: %s", ex)
+            raise HTTPException(
+                status_code=400,
+                detail=f"Couldn't compute certainty with data:\n"
+                f"{final_analysis.logprobs}",
+            ) from ex
+
+    return StagedResponse(
+        explanation=final_analysis,
+        snippets=processed_snippets,
+        response_certainty=certainty,
+    )

{logdetective-1.9.0 → logdetective-2.0.1}/logdetective/server/models.py

@@ -88,6 +88,24 @@ class EmojiHook(BaseModel):
     merge_request: EmojiMergeRequest = Field(default=None)
 
 
+class SnippetAnalysis(BaseModel):
+    """Model of snippet analysis from LLM."""
+
+    text: str = Field(description="Analysis of log snippet contents.")
+
+
+class RatedSnippetAnalysis(SnippetAnalysis):
+    """Model for rated snippet analysis. This model is used to generate
+    json schema for inference with structured output."""
+
+    relevance: int = Field(
+        ge=0,
+        le=100,
+        description="Estimate of likelyhood that snippet contains an error, "
+        "with 0 standing for completely unlikely, 100 for absolutely certain.",
+    )
+
+
 class Explanation(BaseModel):
     """Model of snippet or general log explanation from Log Detective"""
 
@@ -95,6 +113,7 @@ class Explanation(BaseModel):
     logprobs: Optional[List[Dict]] = None
 
     def __str__(self):
+        """Return text of the Explanation"""
         return self.text
 
 
@@ -106,7 +125,7 @@ class AnalyzedSnippet(BaseModel):
     line_number: location of snippet in original log
     """
 
-    explanation: Explanation
+    explanation: SnippetAnalysis | RatedSnippetAnalysis
     text: str
     line_number: int
 
@@ -228,6 +247,7 @@ class ExtractorConfig(BaseModel):
     context: bool = True
     max_clusters: int = 8
     verbose: bool = False
+    max_snippet_len: int = 2000
 
     def __init__(self, data: Optional[dict] = None):
         super().__init__()
@@ -237,6 +257,7 @@ class ExtractorConfig(BaseModel):
         self.context = data.get("context", True)
         self.max_clusters = data.get("max_clusters", 8)
         self.verbose = data.get("verbose", False)
+        self.max_snippet_len = data.get("max_snippet_len", 2000)
 
 
 class GitLabInstanceConfig(BaseModel):  # pylint: disable=too-many-instance-attributes
@@ -439,6 +460,7 @@ class GeneralConfig(BaseModel):
     devmode: bool = False
     sentry_dsn: HttpUrl | None = None
     collect_emojis_interval: int = 60 * 60  # seconds
+    top_k_snippets: int = 0
 
     def __init__(self, data: Optional[dict] = None):
         super().__init__()
@@ -452,6 +474,7 @@ class GeneralConfig(BaseModel):
         self.collect_emojis_interval = data.get(
             "collect_emojis_interval", 60 * 60
         )  # seconds
+        self.top_k_snippets = data.get("top_k_snippets", 0)
 
 
 class Config(BaseModel):

{logdetective-1.9.0 → logdetective-2.0.1}/logdetective/server/server.py

@@ -7,6 +7,7 @@ from typing import Annotated
 from io import BytesIO
 
 import matplotlib
+import matplotlib.figure
 import matplotlib.pyplot
 from fastapi import (
     FastAPI,
@@ -34,21 +35,15 @@ from logdetective.server.database.models.exceptions import (
 
 import logdetective.server.database.base
 
-from logdetective.utils import (
-    compute_certainty,
-    format_snippets,
-    prompt_to_messages,
-)
-
-from logdetective.server.config import SERVER_CONFIG, PROMPT_CONFIG, LOG
+from logdetective.server.config import SERVER_CONFIG, LOG
 from logdetective.server.koji import (
     get_failed_log_from_task as get_failed_log_from_koji_task,
 )
 from logdetective.remote_log import RemoteLog
 from logdetective.server.llm import (
-    mine_logs,
     perform_staged_analysis,
-    submit_text,
+    perfrom_analysis,
+    perform_analyis_stream,
 )
 from logdetective.server.gitlab import process_gitlab_job_event
 from logdetective.server.metric import track_request, add_new_metrics, update_metrics
@@ -157,31 +152,8 @@ async def analyze_log(
     """
     remote_log = RemoteLog(build_log.url, http_session)
     log_text = await remote_log.process_url()
-    log_summary = mine_logs(log_text)
-    log_summary = format_snippets(log_summary)
-    messages = prompt_to_messages(
-        PROMPT_CONFIG.prompt_template.format(log_summary),
-        PROMPT_CONFIG.default_system_prompt,
-        SERVER_CONFIG.inference.system_role,
-        SERVER_CONFIG.inference.user_role,
-    )
-    response = await submit_text(
-        messages,
-        inference_cfg=SERVER_CONFIG.inference,
-    )
-    certainty = 0
 
-    if response.logprobs is not None:
-        try:
-            certainty = compute_certainty(response.logprobs)
-        except ValueError as ex:
-            LOG.error("Error encountered while computing certainty: %s", ex)
-            raise HTTPException(
-                status_code=400,
-                detail=f"Couldn't compute certainty with data:\n{response.logprobs}",
-            ) from ex
-
-    return Response(explanation=response, response_certainty=certainty)
+    return await perfrom_analysis(log_text)
 
 
 @app.post("/analyze/staged", response_model=StagedResponse)
@@ -351,9 +323,7 @@ async def analyze_koji_task(task_id: int, koji_instance_config: KojiInstanceConf
     # Notify any callbacks that the analysis is complete.
     for callback in koji_instance_config.get_callbacks(task_id):
         LOG.info("Notifying callback %s of task %d completion", callback, task_id)
-        asyncio.create_task(
-            send_koji_callback(callback, task_id)
-        )
+        asyncio.create_task(send_koji_callback(callback, task_id))
 
     # Now that it's sent, we can clear the callbacks for this task.
     koji_instance_config.clear_callbacks(task_id)
@@ -398,20 +368,8 @@ async def analyze_log_stream(
     """
     remote_log = RemoteLog(build_log.url, http_session)
     log_text = await remote_log.process_url()
-    log_summary = mine_logs(log_text)
-    log_summary = format_snippets(log_summary)
-    messages = prompt_to_messages(
-        PROMPT_CONFIG.prompt_template.format(log_summary),
-        PROMPT_CONFIG.default_system_prompt,
-        SERVER_CONFIG.inference.system_role,
-        SERVER_CONFIG.inference.user_role,
-    )
     try:
-        stream = submit_text(
-            messages,
-            inference_cfg=SERVER_CONFIG.inference,
-            stream=True,
-        )
+        stream = perform_analyis_stream(log_text)
     except aiohttp.ClientResponseError as ex:
         raise HTTPException(
             status_code=400,
@@ -419,9 +377,6 @@ async def analyze_log_stream(
             f"[{ex.status}] {ex.message}",
         ) from ex
 
-    # we need to figure out a better response here, this is how it looks rn:
-    # b'data: {"choices":[{"finish_reason":"stop","index":0,"delta":{}}],
-    #   "created":1744818071,"id":"chatcmpl-c9geTxNcQO7M9wR...
     return StreamingResponse(stream)
 
 
@@ -711,7 +666,7 @@ async def collect_emoji_task():
             instance.url,
             datetime.datetime.now(datetime.timezone.utc),
         )
-        await collect_emojis(instance.get_connection(), TimePeriod(weeks="54"))
+        await collect_emojis(instance.get_connection(), TimePeriod(weeks=54))
         LOG.info(
             "Collect emoji feedback finished at %s",
             datetime.datetime.now(datetime.timezone.utc),

{logdetective-1.9.0 → logdetective-2.0.1}/logdetective/server/templates/gitlab_full_comment.md.j2

@@ -1,7 +1,9 @@
 The package {{ package }} failed to build, here is a possible explanation why.
 
 Please know that the explanation was provided by AI and may be incorrect.
-In this case, we are {{ certainty }}% certain of the response {{ emoji_face }}.
+{% if certainty > 0 %}
+In this case, we are {{ "%.2f" | format(certainty) }}% certain of the response {{ emoji_face }}.
+{% endif %}
 
 {{ explanation }}
 
@@ -10,7 +12,7 @@ In this case, we are {{ certainty }}% certain of the response {{ emoji_face }}.
 {% for snippet in snippets %}
 <li>
 <b>Line {{ snippet.line_number }}:</b> <code>{{ snippet.text }}</code>
-{{ snippet.explanation }}
+{{ snippet.explanation.text }}
 </li>
 {% endfor %}
 </ul>

{logdetective-1.9.0 → logdetective-2.0.1}/logdetective/server/templates/gitlab_short_comment.md.j2

@@ -1,7 +1,9 @@
 The package {{ package }} failed to build, here is a possible explanation why.
 
 Please know that the explanation was provided by AI and may be incorrect.
-In this case, we are {{ certainty }}% certain of the response {{ emoji_face }}.
+{% if certainty > 0 %}
+In this case, we are {{ "%.2f" | format(certainty) }}% certain of the response {{ emoji_face }}.
+{% endif %}
 
 {{ explanation }}
 

logdetective-2.0.1/logdetective/server/utils.py

@@ -0,0 +1,122 @@
+from typing import List, Tuple
+
+import aiohttp
+from fastapi import HTTPException
+
+from logdetective.constants import SNIPPET_DELIMITER
+from logdetective.extractors import DrainExtractor
+from logdetective.server.config import (
+    LOG,
+    SERVER_CONFIG,
+    SKIP_SNIPPETS_CONFIG,
+)
+from logdetective.server.exceptions import LogDetectiveConnectionError
+from logdetective.server.models import AnalyzedSnippet, RatedSnippetAnalysis
+
+
+def format_analyzed_snippets(snippets: list[AnalyzedSnippet]) -> str:
+    """Format snippets for submission into staged prompt."""
+    summary = f"\n{SNIPPET_DELIMITER}\n".join(
+        [f"[{e.text}] at line [{e.line_number}]: [{e.explanation}]" for e in snippets]
+    )
+    return summary
+
+
+def mine_logs(log: str) -> List[Tuple[int, str]]:
+    """Extract snippets from log text"""
+    extractor = DrainExtractor(
+        verbose=True,
+        context=True,
+        max_clusters=SERVER_CONFIG.extractor.max_clusters,
+        skip_snippets=SKIP_SNIPPETS_CONFIG,
+        max_snippet_len=SERVER_CONFIG.extractor.max_snippet_len
+    )
+
+    LOG.info("Getting summary")
+    log_summary = extractor(log)
+
+    ratio = len(log_summary) / len(log.split("\n"))
+    LOG.debug("Log summary: \n %s", log_summary)
+    LOG.info("Compression ratio: %s", ratio)
+
+    return log_summary
+
+
+def connection_error_giveup(details: dict) -> None:
+    """Too many connection errors, give up.
+    """
+    LOG.error("Too many connection errors, giving up. %s", details["exception"])
+    raise LogDetectiveConnectionError() from details["exception"]
+
+
+def should_we_giveup(exc: aiohttp.ClientResponseError) -> bool:
+    """From backoff's docs:
+
+    > a function which accepts the exception and returns
+    > a truthy value if the exception should not be retried
+    """
+    LOG.info("Should we give up on retrying error %s", exc)
+    return exc.status < 400
+
+
+def we_give_up(details: dict):
+    """Retries didn't work (or we got a different exc)
+    we give up and raise proper 500 for our API endpoint
+    """
+    LOG.error("Last exception: %s", details["exception"])
+    LOG.error("Inference error: %s", details["args"])
+    raise HTTPException(500, "Request to the inference API failed")
+
+
+def select_relevance(snippet: AnalyzedSnippet) -> float:
+    """Retrieve relevance value from structure, if there is one."""
+    if not isinstance(snippet.explanation, RatedSnippetAnalysis):
+        LOG.exception("Only rated snippets can be ordered by relevance.")
+        raise ValueError
+    return snippet.explanation.relevance
+
+
+def select_line_number(explanation: AnalyzedSnippet) -> int:
+    """Returns line number of original snippet."""
+    return explanation.line_number
+
+
+def filter_snippets(
+    processed_snippets: List[AnalyzedSnippet], top_k: int
+) -> List[AnalyzedSnippet]:
+    """Filter snippets according to criteria in config while keeping them ordered by line number.
+    If all snippets recieved the same score, return them all.
+    AnalyzedSnippet objects must have `explanation` attribute set to `RatedSnippetAnalysis`,
+    otherwise raise `ValueError`."""
+
+    if top_k >= len(processed_snippets):
+        LOG.warning(
+            "The `top-k` parameter >= number of original snippets, skipping filtering."
+        )
+        return processed_snippets
+
+    # Sorting invokes `select_relevance` which also tests if objects actually
+    # have the score assigned. Otherwise it raises exception.
+    processed_snippets = sorted(processed_snippets, key=select_relevance, reverse=True)
+
+    # Check for failure mode when all snippets have
+    # the same relevance. In such cases there is no point in filtering
+    # and all snippets are returned.
+    max_relevance = processed_snippets[0].explanation.relevance
+    min_relevance = processed_snippets[-1].explanation.relevance
+
+    LOG.info(
+        "Analyzed snippets sorted. Max relevance: %d Min relevance: %e",
+        max_relevance,
+        min_relevance,
+    )
+    if max_relevance == min_relevance:
+        LOG.warning("All snippets recieved the same rating. Filtering disabled.")
+        return processed_snippets
+
+    processed_snippets = processed_snippets[:top_k]
+
+    # Re-sorting snippets by line number
+    processed_snippets = sorted(processed_snippets, key=select_line_number)
+
+    return processed_snippets

{logdetective-1.9.0 → logdetective-2.0.1}/logdetective/utils.py

@@ -39,7 +39,7 @@ def chunk_continues(text: str, index: int) -> bool:
     return False
 
 
-def get_chunks(text: str) -> Generator[Tuple[int, str], None, None]:
+def get_chunks(text: str, max_len: int = 2000) -> Generator[Tuple[int, str], None, None]:
     """Split log into chunks according to heuristic
     based on whitespace and backslash presence.
     """
@@ -54,7 +54,7 @@ def get_chunks(text: str) -> Generator[Tuple[int, str], None, None]:
         chunk += text[i]
         if text[i] == "\n":
             next_line_number += 1
-            if i + 1 < text_len and chunk_continues(text, i):
+            if i + 1 < text_len and chunk_continues(text, i) and i + 1 < max_len:
                 i += 1
                 continue
             yield (original_line_number, chunk)

{logdetective-1.9.0 → logdetective-2.0.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "logdetective"
-version = "1.9.0"
+version = "2.0.1"
 description = "Log using LLM AI to search for build/test failures and provide ideas for fixing these."
 authors = ["Jiri Podivin <jpodivin@gmail.com>"]
 license = "Apache-2.0"

logdetective-1.9.0/logdetective/server/llm.py

@@ -1,191 +0,0 @@
-import os
-import asyncio
-import random
-from typing import List, Tuple, Union, Dict
-
-import backoff
-from fastapi import HTTPException
-
-import aiohttp
-from openai import AsyncStream
-from openai.types.chat import ChatCompletionChunk
-
-from logdetective.constants import SNIPPET_DELIMITER
-from logdetective.extractors import DrainExtractor
-from logdetective.utils import (
-    compute_certainty,
-    prompt_to_messages,
-)
-from logdetective.server.config import (
-    LOG,
-    SERVER_CONFIG,
-    PROMPT_CONFIG,
-    CLIENT,
-    SKIP_SNIPPETS_CONFIG,
-)
-from logdetective.server.models import (
-    AnalyzedSnippet,
-    InferenceConfig,
-    Explanation,
-    StagedResponse,
-)
-
-
-LLM_CPP_SERVER_TIMEOUT = os.environ.get("LLAMA_CPP_SERVER_TIMEOUT", 600)
-
-
-def format_analyzed_snippets(snippets: list[AnalyzedSnippet]) -> str:
-    """Format snippets for submission into staged prompt."""
-    summary = f"\n{SNIPPET_DELIMITER}\n".join(
-        [
-            f"[{e.text}] at line [{e.line_number}]: [{e.explanation.text}]"
-            for e in snippets
-        ]
-    )
-    return summary
-
-
-def mine_logs(log: str) -> List[Tuple[int, str]]:
-    """Extract snippets from log text"""
-    extractor = DrainExtractor(
-        verbose=True,
-        context=True,
-        max_clusters=SERVER_CONFIG.extractor.max_clusters,
-        skip_snippets=SKIP_SNIPPETS_CONFIG,
-    )
-
-    LOG.info("Getting summary")
-    log_summary = extractor(log)
-
-    ratio = len(log_summary) / len(log.split("\n"))
-    LOG.debug("Log summary: \n %s", log_summary)
-    LOG.info("Compression ratio: %s", ratio)
-
-    return log_summary
-
-
-def should_we_giveup(exc: aiohttp.ClientResponseError) -> bool:
-    """
-    From backoff's docs:
-
-    > a function which accepts the exception and returns
-    > a truthy value if the exception should not be retried
-    """
-    LOG.info("Should we give up on retrying error %s", exc)
-    return exc.status < 400
-
-
-def we_give_up(details: backoff._typing.Details):
-    """
-    retries didn't work (or we got a different exc)
-    we give up and raise proper 500 for our API endpoint
-    """
-    LOG.error("Last exception: %s", details["exception"])
-    LOG.error("Inference error: %s", details["args"])
-    raise HTTPException(500, "Request to the inference API failed")
-
-
-@backoff.on_exception(
-    lambda: backoff.constant([10, 30, 120]),
-    aiohttp.ClientResponseError,
-    max_tries=4,  # 4 tries and 3 retries
-    jitter=lambda wait_gen_value: random.uniform(wait_gen_value, wait_gen_value + 30),
-    giveup=should_we_giveup,
-    raise_on_giveup=False,
-    on_giveup=we_give_up,
-)
-async def submit_text(
-    messages: List[Dict[str, str]],
-    inference_cfg: InferenceConfig,
-    stream: bool = False,
-) -> Union[Explanation, AsyncStream[ChatCompletionChunk]]:
-    """Submit prompt to LLM.
-    inference_cfg: The configuration section from the config.json representing
-    the relevant inference server for this request.
-    log_probs: number of token choices to produce log probs for
-    """
-    LOG.info("Analyzing the text")
-
-    LOG.info("Submitting to /v1/chat/completions endpoint")
-
-    async with inference_cfg.get_limiter():
-        response = await CLIENT.chat.completions.create(
-            messages=messages,
-            max_tokens=inference_cfg.max_tokens,
-            logprobs=inference_cfg.log_probs,
-            stream=stream,
-            model=inference_cfg.model,
-            temperature=inference_cfg.temperature,
-        )
-
-    if isinstance(response, AsyncStream):
-        return response
-    if not response.choices[0].message.content:
-        LOG.error("No response content recieved from %s", inference_cfg.url)
-        raise RuntimeError()
-    if response.choices[0].logprobs and response.choices[0].logprobs.content:
-        logprobs = [e.to_dict() for e in response.choices[0].logprobs.content]
-    else:
-        logprobs = None
-
-    return Explanation(
-        text=response.choices[0].message.content,
-        logprobs=logprobs,
-    )
-
-
-async def perform_staged_analysis(log_text: str) -> StagedResponse:
-    """Submit the log file snippets to the LLM and retrieve their results"""
-    log_summary = mine_logs(log_text)
-
-    # Process snippets asynchronously
-    awaitables = [
-        submit_text(
-            prompt_to_messages(
-                PROMPT_CONFIG.snippet_prompt_template.format(s),
-                PROMPT_CONFIG.snippet_system_prompt,
-                SERVER_CONFIG.inference.system_role,
-                SERVER_CONFIG.inference.user_role,
-            ),
-            inference_cfg=SERVER_CONFIG.snippet_inference,
-        )
-        for s in log_summary
-    ]
-    analyzed_snippets = await asyncio.gather(*awaitables)
-
-    analyzed_snippets = [
-        AnalyzedSnippet(line_number=e[0][0], text=e[0][1], explanation=e[1])
-        for e in zip(log_summary, analyzed_snippets)
-    ]
-    final_prompt = PROMPT_CONFIG.prompt_template_staged.format(
-        format_analyzed_snippets(analyzed_snippets)
-    )
-    messages = prompt_to_messages(
-        final_prompt,
-        PROMPT_CONFIG.staged_system_prompt,
-        SERVER_CONFIG.inference.system_role,
-        SERVER_CONFIG.inference.user_role,
-    )
-    final_analysis = await submit_text(
-        messages,
-        inference_cfg=SERVER_CONFIG.inference,
-    )
-
-    certainty = 0
-
-    if final_analysis.logprobs:
-        try:
-            certainty = compute_certainty(final_analysis.logprobs)
-        except ValueError as ex:
-            LOG.error("Error encountered while computing certainty: %s", ex)
-            raise HTTPException(
-                status_code=400,
-                detail=f"Couldn't compute certainty with data:\n"
-                f"{final_analysis.logprobs}",
-            ) from ex
-
-    return StagedResponse(
-        explanation=final_analysis,
-        snippets=analyzed_snippets,
-        response_certainty=certainty,
-    )