logdetective 3.0.0__tar.gz → 3.1.0__tar.gz

{logdetective-3.0.0 → logdetective-3.1.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: logdetective
-Version: 3.0.0
-Summary: Log using LLM AI to search for build/test failures and provide ideas for fixing these.
+Version: 3.1.0
+Summary: Analyze logs with a template miner and an LLM to discover errors and suggest solutions.
 License: Apache-2.0
 License-File: LICENSE
 Author: Jiri Podivin
@@ -96,11 +96,12 @@ Usage
 
 To analyze a log file, run the script with the following command line arguments:
 - `file` (required): The path or URL of the log file to be analyzed.
-- `--model` (optional, default: "Mistral-7B-Instruct-v0.3-GGUF"): The path or Hugging space name of the language model for analysis. For models from Hugging Face, write them as `namespace/repo_name`. As we are using LLama.cpp we want this to be in the `gguf` format. If the model is already on your machine it will skip the download.
+- `--model` (optional, default: "granite-3.2-8b-instruct-GGUF"): The path or Hugging space name of the language model for analysis. For models from Hugging Face, write them as `namespace/repo_name`. As we are using LLama.cpp we want this to be in the `gguf` format. If the model is already on your machine it will skip the download.
 - `--filename-suffix` (optional, default "Q4_K.gguf"): You can specify which suffix of the file to use. This option is applied when specifying model using the Hugging Face repository.
 - `--n-clusters` (optional, default 8): Number of clusters for Drain to organize log chunks into. This only makes sense when you are summarizing with Drain.
-- `--skip-snippets` Path to patterns for skipping snippets (in YAML).
-- `--prompts PROMPTS` Path to prompt configuration file.
+- `--prompts PROMPTS` (Deprecated, replaced by `--prompts-config`) Path to prompt configuration file.
+- `--prompts-config PROMPTS` Path to prompt configuration file.
+- `--prompt-templates` Path to prompt template dir. Prompts must be valid Jinja templates, and system prompts must include field `system_time`.
 - `--temperature` Temperature for inference.
 - `--skip-snippets` Path to patterns for skipping snippets.
 - `--csgrep` Use csgrep to process the log.
@@ -120,9 +121,15 @@ Examples of using different models. Note the use of `--filename-suffix` (or `-F`
 
 Example of altered prompts:
 
-     cp ~/.local/lib/python3.13/site-packages/logdetective/prompts.yml ~/my-prompts.yml
-     vi ~/my-prompts.yml # edit the prompts there to better fit your needs
-     logdetective https://kojipkgs.fedoraproject.org//work/tasks/3367/131313367/build.log --prompts ~/my-prompts.yml
+    cp -r ~/.local/lib/python3.13/site-packages/logdetective/prompts ~/my-prompts
+    vi ~/my-prompts/system_prompt.j2 # edit the system prompt there to better fit your needs
+    logdetective https://kojipkgs.fedoraproject.org//work/tasks/3367/131313367/build.log --prompt-templates ~/my-prompts
+
+Example of altered prompts (Deprecated):
+
+    cp ~/.local/lib/python3.13/site-packages/logdetective/prompts.yml ~/my-prompts.yml
+    vi ~/my-prompts.yml # edit the prompts there to better fit your needs
+    logdetective https://kojipkgs.fedoraproject.org//work/tasks/3367/131313367/build.log --prompts ~/my-prompts.yml
 
 
 Note that streaming with some models (notably Meta-Llama-3) is broken and can be worked around by `no-stream` option:
@@ -206,7 +213,8 @@ message is reported indicating that the 'check' phase of the rpm build process
 failed with a bad exit status.
 ```
 
-It looks like a wall of text. Similar to any log. The main difference is that here we have the most significant lines of a logfile wrapped in `[ ] : ` and followed by textual explanation of the log text done by mistral 7b.
+It looks like a wall of text. Similar to any log.
+The main difference is that here we have the most significant lines of a logfile wrapped in `[ ] : ` and followed by textual explanation of the log text done by local LLM.
 
 
 Contributing
@@ -374,14 +382,14 @@ Before doing `podman-compose up`, make sure to set `MODELS_PATH` environment var
 ```
 $ export MODELS_PATH=/path/to/models/
 $ ll $MODELS_PATH
--rw-r--r--. 1 tt tt 3.9G apr 10 17:18  mistral-7b-instruct-v0.2.Q4_K_S.gguf
+-rw-r--r--. 1 tt tt 3.9G apr 10 17:18  granite-4.0-h-tiny-Q8_0.gguf
 ```
 
 If the variable is not set, `./models` is mounted inside by default.
 
 Model can be downloaded from [our Hugging Space](https://huggingface.co/fedora-copr) by:
 ```
-$ curl -L -o models/mistral-7b-instruct-v0.3.Q4_K.gguf https://huggingface.co/fedora-copr/Mistral-7B-Instruct-v0.3-GGUF/resolve/main/ggml-model-Q4_K.gguf
+$ curl -L -o models/granite-3.2-8b-instruct-v0.3.Q4_K.gguf https://huggingface.co/fedora-copr/granite-3.2-8b-instruct-GGUF/resolve/main/ggml-model-Q4_K.gguf
 ```
 
 Filtering snippet analysis by relevance
@@ -501,17 +509,38 @@ http GET "localhost:8080/metrics/analyze/requests?weeks=5" > /tmp/plot_weeks.svg
 System Prompts
 --------------
 
-Prompt templates used by Log Detective are stored in the `prompts.yml` file.
+Prompts are defined as Jinja templates and placed in location specified by `--prompt-templates` option of the CLI utility, or `LOGDETECTIVE_PROMPT_TEMPLATES` environment variable of the container service. With further, optional, configuration in the `prompts.yml` configuration file.
+
+All system prompt templates must include place for `system_time` variable.
+
+If `references` list is defined in `prompts.yml`, templates must also include a handling for a list of references.
+
+Example:
+
+```jinja
+{% if references %}
+## References:
+
+    {% for reference in references %}
+    * {{ reference.name }} : {{ reference.link }}
+    {% endfor %}
+{% endif %}
+
+```
+
+*Deprecated:*
+
+*Prompt templates used by Log Detective are stored in the `prompts.yml` file.
 It is possible to modify the file in place, or provide your own.
 In CLI you can override prompt templates location using `--prompts` option,
 while in the container service deployment the `LOGDETECTIVE_PROMPTS` environment variable
-is used instead.
+is used instead.*
 
-Prompts need to have a form compatible with python [format string syntax](https://docs.python.org/3/library/string.html#format-string-syntax)
-with spaces, or replacement fields marked with curly braces, `{}` left for insertion of snippets.
+*Prompts need to have a form compatible with python [format string syntax](https://docs.python.org/3/library/string.html#format-string-syntax)
+with spaces, or replacement fields marked with curly braces, `{}` left for insertion of snippets.*
 
-Number of replacement fields in new prompts, must be the same as in originals.
-Although their position may be different.
+*Number of replacement fields in new prompts, must be the same as in originals.
+Although their position may be different.*
 
 
 Skip Snippets

{logdetective-3.0.0 → logdetective-3.1.0}/README.md

@@ -42,11 +42,12 @@ Usage
 
 To analyze a log file, run the script with the following command line arguments:
 - `file` (required): The path or URL of the log file to be analyzed.
-- `--model` (optional, default: "Mistral-7B-Instruct-v0.3-GGUF"): The path or Hugging space name of the language model for analysis. For models from Hugging Face, write them as `namespace/repo_name`. As we are using LLama.cpp we want this to be in the `gguf` format. If the model is already on your machine it will skip the download.
+- `--model` (optional, default: "granite-3.2-8b-instruct-GGUF"): The path or Hugging space name of the language model for analysis. For models from Hugging Face, write them as `namespace/repo_name`. As we are using LLama.cpp we want this to be in the `gguf` format. If the model is already on your machine it will skip the download.
 - `--filename-suffix` (optional, default "Q4_K.gguf"): You can specify which suffix of the file to use. This option is applied when specifying model using the Hugging Face repository.
 - `--n-clusters` (optional, default 8): Number of clusters for Drain to organize log chunks into. This only makes sense when you are summarizing with Drain.
-- `--skip-snippets` Path to patterns for skipping snippets (in YAML).
-- `--prompts PROMPTS` Path to prompt configuration file.
+- `--prompts PROMPTS` (Deprecated, replaced by `--prompts-config`) Path to prompt configuration file.
+- `--prompts-config PROMPTS` Path to prompt configuration file.
+- `--prompt-templates` Path to prompt template dir. Prompts must be valid Jinja templates, and system prompts must include field `system_time`.
 - `--temperature` Temperature for inference.
 - `--skip-snippets` Path to patterns for skipping snippets.
 - `--csgrep` Use csgrep to process the log.
@@ -66,9 +67,15 @@ Examples of using different models. Note the use of `--filename-suffix` (or `-F`
 
 Example of altered prompts:
 
-     cp ~/.local/lib/python3.13/site-packages/logdetective/prompts.yml ~/my-prompts.yml
-     vi ~/my-prompts.yml # edit the prompts there to better fit your needs
-     logdetective https://kojipkgs.fedoraproject.org//work/tasks/3367/131313367/build.log --prompts ~/my-prompts.yml
+    cp -r ~/.local/lib/python3.13/site-packages/logdetective/prompts ~/my-prompts
+    vi ~/my-prompts/system_prompt.j2 # edit the system prompt there to better fit your needs
+    logdetective https://kojipkgs.fedoraproject.org//work/tasks/3367/131313367/build.log --prompt-templates ~/my-prompts
+
+Example of altered prompts (Deprecated):
+
+    cp ~/.local/lib/python3.13/site-packages/logdetective/prompts.yml ~/my-prompts.yml
+    vi ~/my-prompts.yml # edit the prompts there to better fit your needs
+    logdetective https://kojipkgs.fedoraproject.org//work/tasks/3367/131313367/build.log --prompts ~/my-prompts.yml
 
 
 Note that streaming with some models (notably Meta-Llama-3) is broken and can be worked around by `no-stream` option:
@@ -152,7 +159,8 @@ message is reported indicating that the 'check' phase of the rpm build process
 failed with a bad exit status.
 ```
 
-It looks like a wall of text. Similar to any log. The main difference is that here we have the most significant lines of a logfile wrapped in `[ ] : ` and followed by textual explanation of the log text done by mistral 7b.
+It looks like a wall of text. Similar to any log.
+The main difference is that here we have the most significant lines of a logfile wrapped in `[ ] : ` and followed by textual explanation of the log text done by local LLM.
 
 
 Contributing
@@ -320,14 +328,14 @@ Before doing `podman-compose up`, make sure to set `MODELS_PATH` environment var
 ```
 $ export MODELS_PATH=/path/to/models/
 $ ll $MODELS_PATH
--rw-r--r--. 1 tt tt 3.9G apr 10 17:18  mistral-7b-instruct-v0.2.Q4_K_S.gguf
+-rw-r--r--. 1 tt tt 3.9G apr 10 17:18  granite-4.0-h-tiny-Q8_0.gguf
 ```
 
 If the variable is not set, `./models` is mounted inside by default.
 
 Model can be downloaded from [our Hugging Space](https://huggingface.co/fedora-copr) by:
 ```
-$ curl -L -o models/mistral-7b-instruct-v0.3.Q4_K.gguf https://huggingface.co/fedora-copr/Mistral-7B-Instruct-v0.3-GGUF/resolve/main/ggml-model-Q4_K.gguf
+$ curl -L -o models/granite-3.2-8b-instruct-v0.3.Q4_K.gguf https://huggingface.co/fedora-copr/granite-3.2-8b-instruct-GGUF/resolve/main/ggml-model-Q4_K.gguf
 ```
 
 Filtering snippet analysis by relevance
@@ -447,17 +455,38 @@ http GET "localhost:8080/metrics/analyze/requests?weeks=5" > /tmp/plot_weeks.svg
 System Prompts
 --------------
 
-Prompt templates used by Log Detective are stored in the `prompts.yml` file.
+Prompts are defined as Jinja templates and placed in location specified by `--prompt-templates` option of the CLI utility, or `LOGDETECTIVE_PROMPT_TEMPLATES` environment variable of the container service. With further, optional, configuration in the `prompts.yml` configuration file.
+
+All system prompt templates must include place for `system_time` variable.
+
+If `references` list is defined in `prompts.yml`, templates must also include a handling for a list of references.
+
+Example:
+
+```jinja
+{% if references %}
+## References:
+
+    {% for reference in references %}
+    * {{ reference.name }} : {{ reference.link }}
+    {% endfor %}
+{% endif %}
+
+```
+
+*Deprecated:*
+
+*Prompt templates used by Log Detective are stored in the `prompts.yml` file.
 It is possible to modify the file in place, or provide your own.
 In CLI you can override prompt templates location using `--prompts` option,
 while in the container service deployment the `LOGDETECTIVE_PROMPTS` environment variable
-is used instead.
+is used instead.*
 
-Prompts need to have a form compatible with python [format string syntax](https://docs.python.org/3/library/string.html#format-string-syntax)
-with spaces, or replacement fields marked with curly braces, `{}` left for insertion of snippets.
+*Prompts need to have a form compatible with python [format string syntax](https://docs.python.org/3/library/string.html#format-string-syntax)
+with spaces, or replacement fields marked with curly braces, `{}` left for insertion of snippets.*
 
-Number of replacement fields in new prompts, must be the same as in originals.
-Although their position may be different.
+*Number of replacement fields in new prompts, must be the same as in originals.
+Although their position may be different.*
 
 
 Skip Snippets

{logdetective-3.0.0 → logdetective-3.1.0}/logdetective/constants.py

@@ -4,7 +4,7 @@ in prompts.yaml instead.
 """
 
 # pylint: disable=line-too-long
-DEFAULT_ADVISOR = "fedora-copr/Mistral-7B-Instruct-v0.3-GGUF"
+DEFAULT_ADVISOR = "fedora-copr/granite-3.2-8b-instruct-GGUF"
 
 PROMPT_TEMPLATE = """
 Given following log snippets, and nothing else, explain what failure, if any, occured during build of this package.

{logdetective-3.0.0 → logdetective-3.1.0}/logdetective/logdetective.py

@@ -59,10 +59,18 @@ def setup_args():
     parser.add_argument("-q", "--quiet", action="store_true")
     parser.add_argument(
         "--prompts",
+        "--prompts-config",
         type=str,
         default=f"{os.path.dirname(__file__)}/prompts.yml",
         help="Path to prompt configuration file.",
     )
+    parser.add_argument(
+        "--prompt-templates",
+        type=str,
+        default=f"{os.path.dirname(__file__)}/prompts",
+        help="Path to prompt template dir. Prompts must be valid Jinja templates, \
+              and system prompts must include field `system_time`.",
+    )
     parser.add_argument(
         "--temperature",
         type=float,
@@ -97,7 +105,7 @@ async def run():  # pylint: disable=too-many-statements,too-many-locals,too-many
         log_level = 0
 
     # Get prompts configuration
-    prompts_configuration = load_prompts(args.prompts)
+    prompts_configuration = load_prompts(args.prompts, args.prompt_templates)
 
     logging.basicConfig(stream=sys.stdout)
     LOG.setLevel(log_level)

{logdetective-3.0.0 → logdetective-3.1.0}/logdetective/models.py

@@ -21,26 +21,7 @@ class PromptConfig(BaseModel):
     snippet_system_prompt: str = DEFAULT_SYSTEM_PROMPT
     staged_system_prompt: str = DEFAULT_SYSTEM_PROMPT
 
-    def __init__(self, data: Optional[dict] = None):
-        super().__init__()
-        if data is None:
-            return
-        self.prompt_template = data.get("prompt_template", PROMPT_TEMPLATE)
-        self.snippet_prompt_template = data.get(
-            "snippet_prompt_template", SNIPPET_PROMPT_TEMPLATE
-        )
-        self.prompt_template_staged = data.get(
-            "prompt_template_staged", PROMPT_TEMPLATE_STAGED
-        )
-        self.default_system_prompt = data.get(
-            "default_system_prompt", DEFAULT_SYSTEM_PROMPT
-        )
-        self.snippet_system_prompt = data.get(
-            "snippet_system_prompt", DEFAULT_SYSTEM_PROMPT
-        )
-        self.staged_system_prompt = data.get(
-            "staged_system_prompt", DEFAULT_SYSTEM_PROMPT
-        )
+    references: Optional[list[dict[str, str]]] = None
 
 
 class SkipSnippets(BaseModel):

logdetective-3.1.0/logdetective/prompts/message_template.j2

@@ -0,0 +1,2 @@
+Snippets:
+{}

logdetective-3.1.0/logdetective/prompts/snippet_message_template.j2

@@ -0,0 +1,2 @@
+Snippet:
+{}

logdetective-3.1.0/logdetective/prompts/snippet_system_prompt.j2

@@ -0,0 +1,38 @@
+System time: {{ system_time }}
+
+You are a highly capable expert system specialized in packaging and delivery of software using RPM,
+within the RHEL ecosystem. Your purpose is to help package maintainers diagnose and resolve package build failures.
+You are truthful, concise, and helpful.
+
+## Input processing
+
+You will work with snippets of logs produced during package build.
+These snippets were extracted using data mining algorithm, and may not contain information
+useful for diagnosing the root cause. Snippets without useful information must be disregarded.
+
+## Analysis procedure
+
+1. Provide the snippet with a short explanation.
+2. If the snippet doesn't contain useful information, indicate the fact with a short sentence.
+
+## Examples:
+
+User: "Snippet: RPM build errors:"
+Assistant: "Errors occurred during package build.
+---
+User: "Snippet: Copr build error: Build failed"
+Assistant: "The build in Copr has failed."
+---
+User: "Snippet: /bin/tar: Removing leading `/' from member names"
+Assistant: "This snippet is irrelevant."
+---
+
+{% if references %}
+## References:
+
+When necessary, suggest resources that may be helpful to user.
+
+    {% for reference in references %}
+    * {{ reference.name }} : {{ reference.link }}
+    {% endfor %}
+{% endif %}

logdetective-3.1.0/logdetective/prompts/staged_message_template.j2

@@ -0,0 +1,2 @@
+Snippets:
+{}

logdetective-3.1.0/logdetective/prompts/staged_system_prompt.j2

@@ -0,0 +1,45 @@
+System time: {{ system_time }}
+
+You are a highly capable expert system specialized in packaging and delivery of software using RPM,
+within the RHEL ecosystem. Your purpose is to help package maintainers diagnose and resolve package build failures.
+You are truthful, concise, and helpful.
+
+## Input processing
+
+You will work with snippets of logs produced during package build.
+These snippets were extracted using data mining algorithm, and may not contain information
+useful for diagnosing the root cause. Snippets without useful information must be disregarded.
+
+## Analysis procedure
+
+Analyzed snippets are a format of [X] : [Y], where [X] is a log snippet, and [Y] is the explanation.
+Do not reanalyze the raw log [X].
+
+Snippets are delimited with '================'.
+
+1. Analyze individual snippets, unless they already have analysis attached.
+2. Disregard snippets that do not contain useful information.
+3. Using information from all snippets provide explanation of the issue.
+4. (Optional) Recommend a solution for the package maintainer, only if the cause is clear.
+
+## Examples:
+
+User: "
+    Snippets:
+        ================
+        Snippet No. 1 at line #452:
+            [error: command 'gcc' failed: No such file or directory]: [`gcc` compiler is not available in the build environment]
+        ================
+        Snippet No. 2 at line #452:
+            [Copr build error: Build failed]: [Package build in Copr failed]"
+Assistant: "Package build in Copr failed due to missing `gcc` compiler. Ensure that all build requirements are correctly specified in the spec file."
+
+{% if references %}
+## References:
+
+When necessary, suggest resources that may be helpful to user.
+
+    {% for reference in references %}
+    * {{ reference.name }} : {{ reference.link }}
+    {% endfor %}
+{% endif %}

logdetective-3.1.0/logdetective/prompts/system_prompt.j2

@@ -0,0 +1,57 @@
+System time: {{ system_time }}
+
+You are a highly capable expert system specialized in packaging and delivery of software using RPM,
+within the RHEL ecosystem. Your purpose is to help package maintainers diagnose and resolve package build failures.
+You are truthful, concise, and helpful.
+
+## Input processing
+
+You will work with snippets of logs produced during a failed package build.
+These snippets were extracted using data mining algorithm, and may not contain information
+useful for diagnosing the root cause. Snippets without useful information must be disregarded.
+General error messages, such as failure of commands used during build, are expected.
+
+## Temporal Logic and Causality
+
+Log snippets are typically provided in chronological order. When analyzing multiple snippets
+the first significant error in the log is usually the root cause.
+
+An error occurring at line #500 cannot be caused by an error occurring at line #1000.
+
+Subsequent errors are often side effects of the initial failure. Focus your diagnosis on the primary trigger.
+
+## Analysis procedure
+
+Snippets are provided in order of appearance in the original log, with attached line number,
+and are delimited with '================'.
+Avoid generic or boilerplate recommendations (e.g., "check the logs," "ensure dependencies are met").
+If a specific root cause is identified, the recommendation must directly address that cause.
+
+1. Analyze individual snippets. Do not quote analyzed snippets.
+2. Disregard snippets that do not contain useful information.
+3. Using information from all snippets provide explanation of the issue. Be as specific as possible.
+4. (Optional) Recommend a solution for the package maintainer, only if the cause is clear.
+
+## Examples:
+
+User: "
+    Snippets:
+    Snippet No. 1 at line #452:
+
+    error: command 'gcc' failed: No such file or directory
+    ================
+    Snippet No. 2 at line #560:
+
+    Copr build error: Build failed
+    ================"
+Assistant: "Package build in Copr failed due to missing `gcc` compiler. Ensure that all build requirements are correctly specified in the spec file."
+
+{% if references %}
+## References:
+
+When necessary, suggest resources that may be helpful to user.
+
+    {% for reference in references %}
+    * {{ reference.name }} : {{ reference.link }}
+    {% endfor %}
+{% endif %}

logdetective-3.1.0/logdetective/prompts.py

@@ -0,0 +1,87 @@
+from datetime import datetime, timezone
+from typing import Optional
+from jinja2 import Environment, FileSystemLoader, Template
+
+from logdetective.models import PromptConfig
+
+
+class PromptManager:  # pylint: disable=too-many-instance-attributes
+    """Manages prompts defined as jinja templates"""
+    _tmp_env: Environment
+
+    # Templates for system prompts
+    _default_system_prompt_template: Template
+    _snippet_system_prompt_template: Template
+    _staged_system_prompt_template: Template
+
+    # Templates for messages
+    default_message_template: Template
+    snippet_message_template: Template
+    staged_message_template: Template
+
+    _references: Optional[list[dict[str, str]]] = None
+
+    def __init__(
+        self, prompts_path: str, prompts_configuration: Optional[PromptConfig] = None
+    ) -> None:
+        self._tmp_env = Environment(loader=FileSystemLoader(prompts_path))
+
+        self._default_system_prompt_template = self._tmp_env.get_template(
+            "system_prompt.j2"
+        )
+        self._snippet_system_prompt_template = self._tmp_env.get_template(
+            "snippet_system_prompt.j2"
+        )
+        self._staged_system_prompt_template = self._tmp_env.get_template(
+            "staged_system_prompt.j2"
+        )
+
+        self.default_message_template = self._tmp_env.get_template(
+            "message_template.j2"
+        )
+        self.snippet_message_template = self._tmp_env.get_template(
+            "snippet_message_template.j2"
+        )
+        self.staged_message_template = self._tmp_env.get_template(
+            "staged_message_template.j2"
+        )
+
+        if prompts_configuration:
+            self._references = prompts_configuration.references
+
+    # To maintain backward compatibility with `logdetective.models.PromptConfig`
+    @property
+    def default_system_prompt(self) -> str:
+        """Render system prompt from a template"""
+        return self._default_system_prompt_template.render(
+            system_time=datetime.now(timezone.utc), references=self._references
+        )
+
+    @property
+    def snippet_system_prompt(self) -> str:
+        """Render system prompt from a template"""
+        return self._snippet_system_prompt_template.render(
+            system_time=datetime.now(timezone.utc), references=self._references
+        )
+
+    @property
+    def staged_system_prompt(self) -> str:
+        """Render system prompt from a template"""
+        return self._staged_system_prompt_template.render(
+            system_time=datetime.now(timezone.utc), references=self._references
+        )
+
+    @property
+    def prompt_template(self) -> str:
+        """Render message prompt from the template"""
+        return self.default_message_template.render()
+
+    @property
+    def snippet_prompt_template(self) -> str:
+        """Render message prompt from the template"""
+        return self.snippet_message_template.render()
+
+    @property
+    def prompt_template_staged(self) -> str:
+        """Render message prompt from the template"""
+        return self.staged_message_template.render()

{logdetective-3.0.0 → logdetective-3.1.0}/logdetective/prompts.yml

@@ -88,3 +88,10 @@ staged_system_prompt: |
   You never speculate about package being built or fabricate information.
   If you do not know the answer, you acknowledge the fact and end your response.
   Your responses must be as short as possible.
+
+# Optional references, to be used when constructing prompt from Jinja template
+# references:
+#   - name: Fedora Packaging Guidelines
+#     link: https://docs.fedoraproject.org/en-US/packaging-guidelines/
+#   - name: Mock user documentation
+#     link: https://rpm-software-management.github.io/mock/

{logdetective-3.0.0 → logdetective-3.1.0}/logdetective/server/config.py

@@ -61,7 +61,8 @@ def get_openai_api_client(inference_config: InferenceConfig):
 
 
 SERVER_CONFIG_PATH = os.environ.get("LOGDETECTIVE_SERVER_CONF", None)
-SERVER_PROMPT_PATH = os.environ.get("LOGDETECTIVE_PROMPTS", None)
+SERVER_PROMPT_CONF_PATH = os.environ.get("LOGDETECTIVE_PROMPTS", None)
+SERVER_PROMPT_PATH = os.environ.get("LOGDETECTIVE_PROMPT_TEMPLATES", None)
 # The default location for skip patterns is in the same directory
 # as logdetective __init__.py file.
 SERVER_SKIP_PATTERNS_PATH = os.environ.get(
@@ -70,7 +71,7 @@ SERVER_SKIP_PATTERNS_PATH = os.environ.get(
 )
 
 SERVER_CONFIG = load_server_config(SERVER_CONFIG_PATH)
-PROMPT_CONFIG = load_prompts(SERVER_PROMPT_PATH)
+PROMPT_CONFIG = load_prompts(SERVER_PROMPT_CONF_PATH, SERVER_PROMPT_PATH)
 SKIP_SNIPPETS_CONFIG = load_skip_snippet_patterns(SERVER_SKIP_PATTERNS_PATH)
 
 LOG = get_log(SERVER_CONFIG)

{logdetective-3.0.0 → logdetective-3.1.0}/logdetective/utils.py

@@ -1,10 +1,11 @@
 import logging
 import os
 import subprocess as sp
-from typing import Iterator, List, Dict, Tuple, Generator
+from typing import Iterator, List, Dict, Tuple, Generator, Optional
 from urllib.parse import urlparse
 
 import aiohttp
+from jinja2 import exceptions
 import numpy as np
 import yaml
 
@@ -15,6 +16,7 @@ from llama_cpp import (
 )
 from logdetective.constants import SNIPPET_DELIMITER
 from logdetective.models import PromptConfig, SkipSnippets
+from logdetective.prompts import PromptManager
 from logdetective.remote_log import RemoteLog
 
 LOG = logging.getLogger("logdetective")
@@ -127,7 +129,11 @@ def compute_certainty(probs: List[Dict]) -> float:
 
 
 def process_log(
-    log: str, model: Llama, stream: bool, prompt_templates: PromptConfig, temperature: float
+    log: str,
+    model: Llama,
+    stream: bool,
+    prompt_templates: PromptConfig | PromptManager,
+    temperature: float,
 ) -> CreateChatCompletionResponse | Iterator[CreateChatCompletionStreamResponse]:
     """Processes a given log using the provided language model and returns its summary.
 
@@ -135,20 +141,14 @@ def process_log(
         log (str): The input log to be processed.
         model (Llama): The language model used for processing the log.
         stream (bool): Return output as Iterator.
-        prompt_template (str): Which prompt template to use.
+        prompt_templates (PromptConfig | PromptManager): Prompt templates to use with LLM.
         temperature (float): Temperature parameter for model runtime.
     Returns:
         str: The summary of the given log generated by the language model.
     """
     messages = [
-        {
-            "role": "system",
-            "content": prompt_templates.default_system_prompt
-        },
-        {
-            "role": "user",
-            "content": prompt_templates.prompt_template.format(log)
-        },
+        {"role": "system", "content": prompt_templates.default_system_prompt},
+        {"role": "user", "content": prompt_templates.prompt_template.format(log)},
     ]
 
     response = model.create_chat_completion(
@@ -200,26 +200,35 @@ def format_snippets(snippets: list[str] | list[Tuple[int, str]]) -> str:
         else:
             header = f"Snippet No. {i}:"
             snippet_content = s
-        summary += (
-            f"{header}\n"
-            "\n"
-            f"{snippet_content}\n"
-            f"{SNIPPET_DELIMITER}\n"
-            f"\n"
-        )
+        summary += f"{header}\n\n{snippet_content}\n{SNIPPET_DELIMITER}\n\n"
     return summary
 
 
-def load_prompts(path: str | None) -> PromptConfig:
-    """Load prompts from given yaml file if there is one.
-    Alternatively use defaults."""
-    if path:
+def load_prompts(
+    config_path: Optional[str] = None, template_path: Optional[str] = None
+) -> PromptConfig | PromptManager:
+    """Load prompts from yaml file, and optionally initialize `PromptManager`
+    if provided with path to prompt templates.
+    """
+    configuration = PromptConfig()
+    if config_path:
         try:
-            with open(path, "r") as file:
-                return PromptConfig(yaml.safe_load(file))
+            with open(config_path, "r") as file:
+                configuration = PromptConfig(**yaml.safe_load(file))
         except FileNotFoundError:
-            print("Prompt configuration file not found, reverting to defaults.")
-    return PromptConfig()
+            LOG.error(
+                "Prompt configuration file not found, reverting to defaults.",
+                exc_info=True,
+            )
+    if template_path:
+        try:
+            return PromptManager(template_path, configuration)
+        except exceptions.TemplateError:
+            LOG.error(
+                "Prompt templates couldn't be rendered, reverting to defaults.",
+                exc_info=True,
+            )
+    return configuration
 
 
 def prompt_to_messages(

{logdetective-3.0.0 → logdetective-3.1.0}/logdetective.1.asciidoc

@@ -26,7 +26,7 @@ logdetective - Analyze and summarize log files using LLM and Drain templates
   Show usage description and exit.
 
 *-M* *MODEL*, *--model* *MODEL*::
-  The path to the language model for analysis (if stored locally). You can also specify the model by name based on the repo on Hugging face (see Examples). Repo id must be in the form `'namespace/repo_name'`. As we are using `LLama.cpp` we want this to be in the gguf format. If the model is already on your machine it will skip the download. (optional, default: "fedora-copr/Mistral-7B-Instruct-v0.3-GGUF")
+  The path to the language model for analysis (if stored locally). You can also specify the model by name based on the repo on Hugging face (see Examples). Repo id must be in the form `'namespace/repo_name'`. As we are using `LLama.cpp` we want this to be in the gguf format. If the model is already on your machine it will skip the download. (optional, default: "fedora-copr/granite-3.2-8b-instruct-GGUF")
 
 *-F* *FILENAME_SUFFIX*, *--filename-suffix* *FILENAME_SUFFIX*::
   Define the suffix of the model file name to retrieve from Hugging Face. This option only applies when the model is specified by its Hugging face repo name, and not its path. (default `Q4_K.gguf`)
@@ -44,6 +44,7 @@ logdetective - Analyze and summarize log files using LLM and Drain templates
   Suppress non-essential output.
 
 *--prompts* *PROMPTS_FILE*::
+  (Deprecated replaced by *--prompts-config* and *--prompt-templates*)
   Path to prompt configuration file where you can customize (override default) prompts sent to the LLM. See https://github.com/fedora-copr/logdetective/blob/main/logdetective/prompts.yml for reference.
   +
   Prompts need to have a form compatible with Python format string syntax (see https://docs.python.org/3/library/string.html#format-string-syntax) with spaces, or replacement fields marked with curly braces, `{}` left for insertion of snippets. Number of replacement fields in new prompts must be the same as in original, although their position may be different.
@@ -65,6 +66,10 @@ logdetective - Analyze and summarize log files using LLM and Drain templates
 
     child_exit_code_zero: "Child return code was: 0"
 
+*--prompts-config* *PROMPTS*:: Path to prompt configuration file where you can customize (override default) prompts sent to the LLM and set optional configuration. See https://github.com/fedora-copr/logdetective/blob/main/logdetective/prompts.yml for reference.
+
+*--prompt-templates* *TEMPLATE_DIR*:: Path to prompt template dir. Prompts must be valid Jinja templates, and system prompts must include field *system_time*.
+
 == EXAMPLES
 
 Example usage:

{logdetective-3.0.0 → logdetective-3.1.0}/pyproject.toml

@@ -1,7 +1,7 @@
 [tool.poetry]
 name = "logdetective"
-version = "3.0.0"
-description = "Log using LLM AI to search for build/test failures and provide ideas for fixing these."
+version = "3.1.0"
+description = "Analyze logs with a template miner and an LLM to discover errors and suggest solutions."
 authors = ["Jiri Podivin <jpodivin@gmail.com>"]
 license = "Apache-2.0"
 readme = "README.md"
@@ -10,6 +10,12 @@ include = [
     "logdetective/server/templates/gitlab_comment.md.j2",
     "logdetective/prompts.yml",
     "logdetective.1.asciidoc",
+    "logdetective/prompts/system_prompt.j2",
+    "logdetective/prompts/staged_system_prompt.j2",
+    "logdetective/prompts/snippet_system_prompt.j2",
+    "logdetective/prompts/message_template.j2",
+    "logdetective/prompts/staged_message_template.j2",
+    "logdetective/prompts/snippet_system_prompt.j2",
 ]
 packages = [
     { include = "logdetective" }