data-atlas 0.0.1__py3-none-any.whl

data_atlas-0.0.1.dist-info/METADATA

@@ -0,0 +1,139 @@
+Metadata-Version: 2.4
+Name: data-atlas
+Version: 0.0.1
+Summary: Generate clinical data dictionaries using language models
+Author-email: Raffaele Giancotti <raffaele.giancotti@unical.it>
+License: MIT
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: click>=8.0.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: numpy>=1.20.0
+Requires-Dist: requests>=2.25.0
+Requires-Dist: ollama>=0.1.0
+Requires-Dist: torch>=2.0.0
+Requires-Dist: transformers>=4.20.0
+Requires-Dist: sentence-transformers>=2.0.0
+Requires-Dist: scikit-learn>=1.0.0
+Requires-Dist: reportlab>=3.6.0
+Requires-Dist: tqdm>=4.60.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Dynamic: license-file
+
+# Data Dictionary Generator
+
+A Python package to automatically generate data dictionaries for clinical datasets using a large language model (LLM) via Ollama. This tool takes in dataset files (CSV format), processes them, and generates descriptions for each column in the dataset, as well as other metadata like data types, sample data, table descriptions and some quality information such as missing values, outliers and redundant values. It is also able to find relationships between tables and columns.
+
+## Features
+### Metadata Generation
+- **Column Descriptions**: AI-generated clinical context for each field
+- **Table Summaries**: Dataset-level documentation
+- **Smart Sampling**: Automatic data type detection with sample values
+
+### Quality Analysis
+- Missing value statistics
+- Outlier detection (numeric fields)
+- Duplicate row/column identification
+
+### Advanced Capabilities
+- **Semantic Relationship Detection**: Finds connected columns across tables
+- **Multi-Format Outputs**: JSON, Markdown
+- **Custom LLM Integration**: Supports any Ollama, OpenAI and Google Gemini models
+
+## Requirements
+
+Make sure you have Python 3.8+ installed. All dependencies are automatically installed when you install the package.
+
+## Installation
+
+Clone the repository:
+
+```bash
+git clone https://github.com/rafgia/data-dictionary-generator.git
+cd code
+```
+
+Install the package:
+
+```bash
+pip install -e .
+```
+
+## Usage
+
+### Command-line Interface
+
+Once the package is installed, you can use the command line to generate metadata for your dataset(s).
+
+To run the tool, use the following command:
+
+```bash
+python generate_dictionary.py <folder_path> --output-dir <output_dir> --model llama3.1
+```
+
+#### Parameters:
+- `<folder_path>`: The path to the folder containing your CSV files.
+- `<output-dir>`: The name of the path where the metadata will be saved.
+- `--model`: Specify the Ollama or OpenAI model to use for generating metadata.
+
+### Example
+
+1. **Prepare your data files**:
+   Place all your CSV files (representing tables in your dataset) in a folder, e.g., `data/MIMIC`.
+
+2. **Run the generator**:
+
+```bash
+python -m generate_dictionary.py data/MIMIC --output-dir output --model gpt-4o-mini
+```
+
+This will generate metadata for each column in the dataset and save it to a JSON file and a Markdown summary for the dictionary, and a .png plot and a markdown summary for the detected relationships.
+
+### Sample Output
+
+For each **table** in your dataset, the following metadata will be generated:
+- **Table Name**: The name of the table (CSV filename).
+- **Number of Rows**: The total number of rows in the table.
+- **Number of Columns**: The total number of columns in the table.
+- **Table Description**: A generated description of what the table contains.
+
+For each **column** in your dataset:
+- **Column Name**: The name of the column.
+- **Sample Data**: A sample of 5 data points from the column.
+- **Data Type**: The inferred data type (e.g., integer, float, string).
+- **Column Description**: A description generated by the model for the column.
+
+### Example of generated metadata:
+
+
+## Troubleshooting
+
+### 1. If you encounter an error such as `model not found`, make sure you have set up Ollama correctly and the model is available.
+- Ensure that you can manually run the model using `ollama run llama3.1` from the command line before using it in the Python script.
+
+### 2. If the dependencies are not installing, make sure you're using the correct Python version and have all required libraries listed in `pyproject.toml`.
+
+### 3. If the dataset is very large, consider breaking it down into smaller CSV files for more efficient processing.
+
+## Contributing
+
+If you would like to contribute to this project, feel free to fork the repository and submit a pull request. Make sure to add tests and document any new features.
+
+### To contribute:
+1. Fork the repository.
+2. Create a new branch (`git checkout -b feature-name`).
+3. Make your changes and commit them (`git commit -am 'Add new feature'`).
+4. Push to the branch (`git push origin feature-name`).
+5. Open a pull request.
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+---
+
+## Author
+
+Raffaele Giancotti

data_atlas-0.0.1.dist-info/RECORD

@@ -0,0 +1,15 @@
+data_atlas-0.0.1.dist-info/licenses/LICENSE,sha256=Fs1PK6wJ87X7i9kIGoRcHOm_HrAFazHyD-yCYZrtvnk,1096
+data_dictionary_generator/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+data_dictionary_generator/column_stats.py,sha256=o3BA1R6o9-LJQKvDbnlvL8HmcdmwaDr0QAGm2a2klZA,1736
+data_dictionary_generator/data_type.py,sha256=i1_WaIkSw_oGnNYSJ8DQ7TGansJS9P9exTbJ1Bzg3-k,2057
+data_dictionary_generator/description_generator.py,sha256=W6ZEoNAf9lludseReSZyFFDJcCIIa5pQBd5M7eoSb58,13487
+data_dictionary_generator/generate_dictionary.py,sha256=sXkpSkEaeMv28O9T2WfUwmfFduBhuH1oHt3TBRa1U0Y,5506
+data_dictionary_generator/load_dataset.py,sha256=33zbTfXve83FfWw1sQgu_D2I-FrFz5ZUm8HFDaSPRWI,1165
+data_dictionary_generator/metadata_extraction.py,sha256=q1M22qAxWERAmtFDvBi3FSP2f5GFlOAW07olUkyPy3s,7650
+data_dictionary_generator/pattern_extraction.py,sha256=yCCSzbWbYH4w4nAgMJfGhxG9xg-UV1C6M5duyucCpkc,4196
+data_dictionary_generator/relationships.py,sha256=-lu4m-yM1_wRdGaYNmDAIJbGGDBa28yscCbj7pmTEM4,11436
+data_atlas-0.0.1.dist-info/METADATA,sha256=6e61nz0lzDevqqstDa_fvIo8Yb-lwaLwDPJL6pNNr0w,5092
+data_atlas-0.0.1.dist-info/WHEEL,sha256=aeYiig01lYGDzBgS8HxWXOg3uV61G9ijOsup-k9o1sk,91
+data_atlas-0.0.1.dist-info/entry_points.txt,sha256=0bveK9ODg6B1uqanTxwVSCppNgdCETz8JCter0r0cLg,91
+data_atlas-0.0.1.dist-info/top_level.txt,sha256=toz52AXzeut2aQUPFNKxYmD8aZf-2-BBY4ZHLGLrgAc,26
+data_atlas-0.0.1.dist-info/RECORD,,

data_atlas-0.0.1.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (82.0.1)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

data_atlas-0.0.1.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+generate-dictionary = data_dictionary_generator.generate_dictionary:main

data_atlas-0.0.1.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Raffaele Giancotti
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data_atlas-0.0.1.dist-info/top_level.txt

@@ -0,0 +1 @@
+data_dictionary_generator

data_dictionary_generator/column_stats.py

@@ -0,0 +1,56 @@
+import pandas as pd
+import numpy as np
+from collections import Counter
+from data_type import infer_data_type
+
+def compute_column_stats(series: pd.Series, sample_size: int = 5000) -> dict:
+    """
+    Compute statistics for a column, based on inferred data type.
+    """
+
+    if series.empty:
+        return None
+
+    non_null_series = series.dropna()
+    if non_null_series.empty:
+        return None
+
+    if len(non_null_series) > sample_size:
+        non_null_series = non_null_series.sample(sample_size, random_state=42)
+
+    inferred_types = non_null_series.apply(infer_data_type)
+    col_type = inferred_types.value_counts().idxmax()
+
+    stats = {}
+
+    if col_type in ["integer", "float"]:
+        numeric = pd.to_numeric(non_null_series, errors="coerce").dropna()
+        if not numeric.empty:
+            stats = {
+                "min": float(numeric.min()),
+                "max": float(numeric.max()),
+                "mean": float(numeric.mean()),
+                "median": float(numeric.median()),
+                "std": float(numeric.std())
+            }
+
+    elif col_type == "datetime":
+        dates = pd.to_datetime(non_null_series, errors="coerce", format="mixed").dropna()
+        if not dates.empty:
+            stats = {
+                "min_date": dates.min().isoformat(),
+                "max_date": dates.max().isoformat()
+            }
+
+    elif col_type in ["string", "boolean"]:
+        counts = Counter(non_null_series.astype(str))
+        top_values = counts.most_common(10)
+        stats = {
+            "top_values": top_values,
+            "unique_count": len(counts)
+        }
+
+    else:
+        return None
+
+    return stats if stats else None

data_dictionary_generator/data_type.py

@@ -0,0 +1,66 @@
+from dateutil.parser import parse as dateparse
+import pandas as pd
+import re
+
+_date_pattern = re.compile(
+    r"(\d{4}[-/]\d{2}[-/]\d{2})"           # 2024-01-31
+    r"|(\d{2}[-/]\d{2}[-/]\d{4})"           # 31-01-2024
+    r"|(\d{4}[-/]\d{2}[-/]\d{2}T\d{2}:\d{2})"  # 2024-01-31T12:00
+)
+
+def infer_data_type(value) -> str:
+    if pd.isna(value):
+        return "null"
+    if isinstance(value, bool):
+        return "boolean"
+    if isinstance(value, int):
+        return "integer"
+    if isinstance(value, float):
+        return "float"
+    if not isinstance(value, str):
+        return "string"
+    v = value.strip()
+    if v == "":
+        return "null"
+    lower = v.lower()
+    if lower in ("true", "false", "yes", "no", "y", "n", "t", "f"):
+        return "boolean"
+    temp_v = v.replace(",", "")
+    has_sign = temp_v.startswith(("+", "-"))
+    numeric_part = temp_v[1:] if has_sign else temp_v
+    if "+" in numeric_part or "-" in numeric_part:
+        return "string"
+    if numeric_part.isdigit():
+        return "integer"
+    try:
+        if "." in numeric_part:
+            if numeric_part.replace(".", "", 1).isdigit():
+                return "float"
+        float(temp_v)
+        return "float"
+    except ValueError:
+        pass
+
+    if _date_pattern.search(v):
+        try:
+            dateparse(v, fuzzy=False)
+            return "datetime"
+        except (ValueError, OverflowError):
+            pass
+
+    return "string"
+
+def infer_column_data_type(series: pd.Series, sample_size: int = 100) -> str:
+    """
+    Infer the most likely data type of a column by sampling non-null values
+    and selecting the most frequent inferred type.
+    """
+    non_null_series = series.dropna()
+    if non_null_series.empty:
+        return "null"
+    sample = non_null_series.sample(
+        min(sample_size, len(non_null_series)), random_state=42
+    )
+    inferred_types = sample.apply(infer_data_type)
+    most_common = inferred_types.value_counts().idxmax()
+    return most_common

data_dictionary_generator/description_generator.py

@@ -0,0 +1,331 @@
+import re
+import os
+import requests
+from typing import Optional
+import logging
+from pydantic import BaseModel, Field
+from metadata_extraction import DatasetMeta, TableMeta, ColumnMeta
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+OPENAI_MODELS = ["gpt-4o", "gpt-4o-mini", "gpt-4-turbo", "gpt-4", "gpt-3.5-turbo"]
+OLLAMA_URL = "http://localhost:11434"
+
+DATASET_PROMPT_TEMPLATE = """
+You are an expert in dataset documentation. 
+Generate a short, precise, and factual description of the dataset.
+Do NOT write introductions, explanations, examples, or subjective opinions.
+Do NOT repeat the dataset name or make assumptions.
+
+Dataset overview:
+- Domain: {{domain}}
+- Tables: {{table_list}}
+- Row counts per table: {{table_row_counts}}
+- Time coverage: {{time_range}}
+
+Output:
+- A single sentence (max 15 words) describing the dataset’s content, purpose, and domain.
+"""
+
+TABLE_PROMPT_TEMPLATE = """
+You are an expert in data documentation. 
+Generate a concise and precise description of the table.
+Do NOT give examples, assumptions, or repeat the table name.
+Use only the provided metadata.
+
+Dataset context:
+- Name: {{dataset_name}}
+- Domain: {{dataset_domain}}
+- Description: {{dataset_description}}
+
+Table metadata:
+- Name: {{table_name}}
+- Columns (name and type): {{column_list_with_types}}
+- Number of rows: {{n_rows}}
+- Sample rows: {{sample_rows}}
+
+Output:
+- One short sentence (max 15 words) describing what the table represents and its content.
+- Include factual details only.
+"""
+
+COLUMN_PROMPT_TEMPLATE = """
+You are an expert in data documentation and metadata analysis.
+Generate a precise, factual, single-sentence description of the column.
+
+Constraints:
+1. Do NOT speculate, give examples, or repeat column/table names.
+2. Do NOT repeat null percentage, distinct counts, or stats summary in the text.
+3. Do NOT start descriptions with "Represents", "Indicates", "Contains", or "Column". Start directly with the subject (e.g., "Patient age..." instead of "Indicates patient age...").
+
+Goals:
+- Capture the unit of measurements (e.g., hours, kg, ml) if implied by sample values.
+- Use the stats only to infer if the data is categorical or continuous, then write the description accordingly.
+
+Dataset context:
+- Name: {{dataset_name}}
+- Domain: {{dataset_domain}}
+- Description: {{dataset_description}}
+
+Table context:
+- Name: {{table_name}}
+- Description: {{table_description}}
+
+Column metadata:
+- Name: {{column_name}}
+
+
+Output:
+- One factual sentence (max 15 words).
+"""
+
+def _calculate_null_percent(col_meta: ColumnMeta) -> str:
+    if col_meta.total_rows == 0:
+        return "N/A"
+    percent = (col_meta.null_count / col_meta.total_rows) * 100
+    return f"{percent:.1f}%"
+
+def _format_column_stats(col_meta: ColumnMeta) -> str:
+    stats = col_meta.stats
+    if not stats or stats.get("type") is None:
+        return f"Unique values: {stats.get('unique_count', 'N/A')}" if stats else "N/A"
+
+    if stats["type"] == "numeric":
+        return (f"numerical: [min: {stats.get('min', 'N/A')}, max: {stats.get('max', 'N/A')}, "
+                f"mean: {stats.get('mean', 'N/A')}, median: {stats.get('median', 'N/A')}]")
+    elif stats["type"] == "datetime":
+        return f"datetime range: [{stats.get('min_date', 'N/A')} to {stats.get('max_date', 'N/A')}]"
+    elif stats["type"] == "categorical":
+        top_vals = stats.get('top_values', [])
+        if top_vals:
+            top_5 = ", ".join([str(v[0]) for v in top_vals[:5]])
+            return f"categorical: top_5_values={{{top_5}}}"
+        return f"categorical: unique_count={stats.get('unique_count', 'N/A')}"
+    return "N/A"
+
+def _format_sample_rows(table_meta: TableMeta) -> str:
+    if not table_meta.sample_rows:
+        return "N/A"
+    return "\n".join([str(row) for row in table_meta.sample_rows])
+
+def generate_dataset_description(dataset_meta: DatasetMeta, model: str, llm_function) -> str:
+    table_counts = {t_name: t_meta.row_count for t_name, t_meta in dataset_meta.tables.items()}
+    time_range = (f"{dataset_meta.time_period_covered.get('start')} to "
+                  f"{dataset_meta.time_period_covered.get('end')}") if dataset_meta.time_period_covered else "N/A"
+
+    prompt = DATASET_PROMPT_TEMPLATE.replace("{{domain}}", dataset_meta.domain_covered or "Unspecified")\
+                                    .replace("{{table_list}}", ", ".join(dataset_meta.list_of_tables))\
+                                    .replace("{{table_row_counts}}", str(table_counts))\
+                                    .replace("{{time_range}}", time_range)
+
+    description = llm_function(prompt, model)
+    return description or f"Dataset containing {len(dataset_meta.list_of_tables)} tables covering {dataset_meta.domain_covered or 'unspecified domain'}."
+
+def generate_table_description(table_meta: TableMeta, dataset_name: str, dataset_description: str, dataset_domain: str, model: str, llm_function) -> str:
+    col_list_types = ", ".join([f"{col_name} ({meta.data_type})" for col_name, meta in table_meta.columns.items()])
+    sample_rows = _format_sample_rows(table_meta)
+
+    prompt = TABLE_PROMPT_TEMPLATE.replace("{{dataset_name}}", dataset_name)\
+                                  .replace("{{dataset_domain}}", dataset_domain)\
+                                  .replace("{{dataset_description}}", dataset_description)\
+                                  .replace("{{table_name}}", table_meta.table_name)\
+                                  .replace("{{column_list_with_types}}", col_list_types)\
+                                  .replace("{{n_rows}}", str(table_meta.row_count))\
+                                  .replace("{{sample_rows}}", sample_rows)
+
+    description = llm_function(prompt, model)
+    return description or f"Table {table_meta.table_name} with {table_meta.number_of_columns} columns and {table_meta.row_count} rows."
+
+def generate_single_column_description(column_meta: ColumnMeta, table_name: str, table_description: str, dataset_name: str, dataset_description: str, dataset_domain: str, model: str, llm_function) -> str:
+    null_percent = _calculate_null_percent(column_meta)
+    stats_summary = _format_column_stats(column_meta)
+    samples_str = ", ".join([str(s) for s in column_meta.sample_values]) if column_meta.sample_values else "N/A"
+
+    type_str = column_meta.semantic_type if column_meta.semantic_type else column_meta.data_type
+    if column_meta.is_primary_key:
+        type_str += " (Primary Key)"
+    if column_meta.foreign_key_to:
+        type_str += f" (Foreign Key to {column_meta.foreign_key_to})"
+
+    prompt = COLUMN_PROMPT_TEMPLATE.replace("{{dataset_name}}", dataset_name)\
+                                   .replace("{{dataset_domain}}", dataset_domain)\
+                                   .replace("{{dataset_description}}", dataset_description)\
+                                   .replace("{{table_name}}", table_name)\
+                                   .replace("{{table_description}}", table_description)\
+                                   .replace("{{column_name}}", column_meta.column_name)\
+                                   .replace("{{type}}", type_str)\
+                                   .replace("{{samples}}", samples_str)\
+                                   .replace("{{unique}}", str(column_meta.unique_count))\
+                                   .replace("{{null_percent}}", null_percent)\
+                                   .replace("{{stats_summary}}", stats_summary)
+
+    response = llm_function(prompt, model)
+    return response if response else f"Column of type {type_str}."
+
+def _post_process_response(response_text: str) -> str:
+    """Cleans LLM response and strictly enforces word count limit."""
+    if not response_text:
+        return ""
+        
+    clean_response = re.sub(r"<think>.*?</think>", "", response_text, flags=re.DOTALL)
+    clean_response = re.sub(r"```.*?```", "", clean_response, flags=re.DOTALL)
+    clean_response = re.sub(r"[*_#>`\"']", "", clean_response)
+    clean_response = re.sub(r"\s+", " ", clean_response).strip()
+    words = clean_response.split()
+    if len(words) > 15:
+        clean_response = " ".join(words[:15])
+        
+    return clean_response
+
+COST_TRACKER = {
+    "total_cost": 0.0,
+    "models": {}
+}
+
+def update_cost(model: str, cost: float):
+    COST_TRACKER["total_cost"] += cost
+    if model not in COST_TRACKER["models"]:
+        COST_TRACKER["models"][model] = {"calls": 0, "cost": 0.0}
+    COST_TRACKER["models"][model]["calls"] += 1
+    COST_TRACKER["models"][model]["cost"] += cost
+
+def get_cost_summary() -> dict:
+    """
+    Returns the entire cost summary.
+    """
+    return COST_TRACKER
+
+def _run_openai_model(prompt: str, model: str) -> Optional[str]:
+    """
+    Call OpenAI for description generation with incremental cost tracking.
+    """
+    try:
+        api_key = os.environ.get("OPENAI_API_KEY")
+        if not api_key:
+            raise ValueError("OPENAI_API_KEY environment variable not set")
+
+        from openai import OpenAI
+        client = OpenAI(api_key=api_key)
+        response = client.chat.completions.create(
+            model=model,
+            messages=[
+                {
+                    "role": "system",
+                    "content": (
+                        "You are a precise metadata expert. "
+                        "You write extremely concise, factual descriptions based ONLY on provided context. "
+                        "Do not use filler words. Strictly adhere to word limits (max 15 words)."
+                    )
+                },
+                {"role": "user", "content": prompt}
+            ],
+            temperature=0.1,
+            max_tokens=50,
+        )
+
+        response_text = response.choices[0].message.content.strip()
+
+        # Token usage
+        usage = getattr(response, "usage", None)
+        if usage:
+            prompt_tokens = usage.prompt_tokens
+            completion_tokens = usage.completion_tokens
+        else:
+            prompt_tokens = completion_tokens = 0
+
+        # Pricing table (per 1k tokens)
+        model_costs = {
+            "gpt-4": (0.03, 0.06),
+            "gpt-4-turbo": (0.01, 0.03),
+            "gpt-4o": (0.005, 0.015),
+            "gpt-4o-mini": (0.00015, 0.0006),
+            "gpt-3.5-turbo": (0.0005, 0.0015),
+        }
+        prompt_cost_per_1k, completion_cost_per_1k = model_costs.get(model, (0.005, 0.015))
+
+        # Cost of this single call
+        cost = (
+            (prompt_tokens / 1000) * prompt_cost_per_1k +
+            (completion_tokens / 1000) * completion_cost_per_1k
+        )
+
+        # Incremental cost update
+        update_cost(model, cost)
+
+        logger.info(
+            f"[COST] Model={model} | Prompt={prompt_tokens} | Completion={completion_tokens} "
+            f"| This call=${cost:.6f} | Total=${COST_TRACKER['total_cost']:.6f}"
+        )
+
+        return _post_process_response(response_text)
+
+    except Exception as e:
+        logger.error(f"OpenAI API error for model {model}: {e}")
+        return None
+    
+def _run_gemini_model(prompt: str, model: str) -> Optional[str]:
+    """
+    Runs Google Gemini models (via google-generativeai).
+    """
+    try:
+        api_key = os.environ.get("GEMINI_API_COST_KEY")
+        if not api_key:
+            raise ValueError("GEMINI_API_KEY environment variable not set")
+
+        import google.generativeai as genai
+        genai.configure(api_key=api_key)
+        gem_model = genai.GenerativeModel(model)
+        response = gem_model.generate_content(prompt)
+        response_text = response.text.strip() if hasattr(response, "text") else ""
+        return _post_process_response(response_text)
+    except Exception as e:
+        logger.error(f"Gemini API error for model {model}: {e}")
+        return None
+
+def run_ollama_model(prompt: str, model: str) -> Optional[str]:
+    """
+    Call local Ollama API for concise descriptions.
+    """
+    API_ENDPOINT = f"{OLLAMA_URL}/api/generate"
+    try:
+        num_predict = 50
+        
+        payload = {
+            "model": model,
+            "prompt": prompt,
+            "stream": False,
+            "options": {
+                "temperature": 0.1,
+                "num_predict": num_predict
+            }
+        }
+        
+        response = requests.post(API_ENDPOINT, json=payload)
+        response.raise_for_status()
+        
+        response_data = response.json()
+        raw_text = response_data.get("response", "").strip()
+        return _post_process_response(raw_text)
+
+    except requests.exceptions.RequestException as e:
+        logger.error(f"Ollama API call failed for model {model}: {e}")
+        return None
+
+
+def run_llm_dispatcher(prompt: str, model: str) -> Optional[str]:
+    """
+    Routes the prompt to the correct LLM implementation (OpenAI, Gemini or Ollama).
+    """
+    model_lower = model.lower()
+
+    # OpenAI Models
+    if model_lower in [m.lower() for m in OPENAI_MODELS] or model_lower.startswith("gpt-"):
+        return _run_openai_model(prompt, model)
+
+    # Google Gemini Models
+    if model_lower.startswith("gemini"):
+        return _run_gemini_model(prompt, model)
+
+    # Local Ollama
+    return run_ollama_model(prompt, model)