maltopic 1.0.0__tar.gz

maltopic-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Yash Sharma
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

maltopic-1.0.0/PKG-INFO

@@ -0,0 +1,100 @@
+Metadata-Version: 2.3
+Name: maltopic
+Version: 1.0.0
+Summary: A multi-agent LLM topic modeling library.
+License: MIT
+Keywords: topic modeling,LLM,multi-agent,survey analysis,data enrichment
+Author: Yash Sharma
+Author-email: yash91sharma@gmail.com
+Requires-Python: >=3.12,<4.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Dist: openai (>=1.79.0,<2.0.0)
+Requires-Dist: pandas (>=2.2.3,<3.0.0)
+Requires-Dist: tqdm (>=4.67.1,<5.0.0)
+Project-URL: Homepage, https://github.com/yash91sharma/MALTopic-py
+Project-URL: Repository, https://github.com/yash91sharma/MALTopic-py
+Description-Content-Type: text/markdown
+
+# MALTopic: Multi-Agent LLM Topic Modeling Library
+
+MALTopic is a powerful library designed for topic modeling using a multi-agent approach. It leverages the capabilities of large language models (LLMs) to enhance the analysis of survey responses by integrating structured and unstructured data.
+
+MALTopic as a research paper was published in 2025 World AI IoT Congress. Links here.
+
+## Features
+
+- **Multi-Agent Framework**: Decomposes topic modeling into specialized tasks executed by individual LLM agents.
+- **Data Enrichment**: Enhances textual responses using structured and categorical survey data.
+- **Latent Theme Extraction**: Extracts meaningful topics from enriched responses.
+- **Topic Deduplication**: Refines and consolidates identified topics for better interpretability.
+
+## Installation
+
+To install the MALTopic library, you can use pip:
+
+```bash
+pip install maltopic
+```
+
+## Usage
+
+To use the MALTopic library, you need to initialize the main class with your API key and model name. You can choose between different LLMs such as OpenAI, Google Gemini (not supported yet), or Llama (not supported yet).
+
+```python
+from maltopic import MALTopic
+
+# Initialize the MALTopic class
+mal_topic = client = MALTopic(
+    api_key="your_api_key",
+    default_model_name="gpt-4.1-nano",
+    llm_type="openai",
+)
+
+enriched_df = client.enrich_free_text_with_structured_data(
+        survey_context="context about survey, why, how of it...",
+        free_text_column="column_1",
+        structured_data_columns=["columns_2", "column_3"],
+        df=df,
+        examples=["free text response, category 1 -> free text response with additional context", "..."], # optional
+    )
+
+topics = client.generate_topics(
+        topic_mining_context="context about what kind of topics you want to mine",
+        df=enriched_df,
+        enriched_column="column_1" + "_enriched", # MALTopic adds _enriched as the suffix.
+    )
+
+print(topics)
+```
+
+## Agents
+
+- **Enrichment Agent**: Enhances free-text responses using structured data.
+- **Topic Modeling Agent**: Extracts latent themes from enriched responses.
+- **Deduplication Agent**: Refines and consolidates the extracted topics. (not supported yet)
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a pull request or open an issue for any enhancements or bug fixes.
+
+## License
+
+This project is licensed under the MIT License. See the LICENSE file for more details.
+
+## Citation
+
+If you use MALTopic in your research, please cite:
+
+```bibtex
+@software{Sharma2025maltopic,
+  author = {Sharma, Yash},
+  title = {MALTopic: A library for topic modeling},
+  year = {2025},
+  url = {https://github.com/yash91sharma/MALTopic-py}
+}
+

maltopic-1.0.0/README.md

@@ -0,0 +1,77 @@
+# MALTopic: Multi-Agent LLM Topic Modeling Library
+
+MALTopic is a powerful library designed for topic modeling using a multi-agent approach. It leverages the capabilities of large language models (LLMs) to enhance the analysis of survey responses by integrating structured and unstructured data.
+
+MALTopic as a research paper was published in 2025 World AI IoT Congress. Links here.
+
+## Features
+
+- **Multi-Agent Framework**: Decomposes topic modeling into specialized tasks executed by individual LLM agents.
+- **Data Enrichment**: Enhances textual responses using structured and categorical survey data.
+- **Latent Theme Extraction**: Extracts meaningful topics from enriched responses.
+- **Topic Deduplication**: Refines and consolidates identified topics for better interpretability.
+
+## Installation
+
+To install the MALTopic library, you can use pip:
+
+```bash
+pip install maltopic
+```
+
+## Usage
+
+To use the MALTopic library, you need to initialize the main class with your API key and model name. You can choose between different LLMs such as OpenAI, Google Gemini (not supported yet), or Llama (not supported yet).
+
+```python
+from maltopic import MALTopic
+
+# Initialize the MALTopic class
+mal_topic = client = MALTopic(
+    api_key="your_api_key",
+    default_model_name="gpt-4.1-nano",
+    llm_type="openai",
+)
+
+enriched_df = client.enrich_free_text_with_structured_data(
+        survey_context="context about survey, why, how of it...",
+        free_text_column="column_1",
+        structured_data_columns=["columns_2", "column_3"],
+        df=df,
+        examples=["free text response, category 1 -> free text response with additional context", "..."], # optional
+    )
+
+topics = client.generate_topics(
+        topic_mining_context="context about what kind of topics you want to mine",
+        df=enriched_df,
+        enriched_column="column_1" + "_enriched", # MALTopic adds _enriched as the suffix.
+    )
+
+print(topics)
+```
+
+## Agents
+
+- **Enrichment Agent**: Enhances free-text responses using structured data.
+- **Topic Modeling Agent**: Extracts latent themes from enriched responses.
+- **Deduplication Agent**: Refines and consolidates the extracted topics. (not supported yet)
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a pull request or open an issue for any enhancements or bug fixes.
+
+## License
+
+This project is licensed under the MIT License. See the LICENSE file for more details.
+
+## Citation
+
+If you use MALTopic in your research, please cite:
+
+```bibtex
+@software{Sharma2025maltopic,
+  author = {Sharma, Yash},
+  title = {MALTopic: A library for topic modeling},
+  year = {2025},
+  url = {https://github.com/yash91sharma/MALTopic-py}
+}

maltopic-1.0.0/pyproject.toml

@@ -0,0 +1,30 @@
+[tool.poetry]
+name = "maltopic"
+version = "1.0.0"
+description = "A multi-agent LLM topic modeling library."
+authors = ["Yash Sharma <yash91sharma@gmail.com>"]
+license = "MIT"
+readme = "README.md"
+homepage = "https://github.com/yash91sharma/MALTopic-py"
+repository = "https://github.com/yash91sharma/MALTopic-py"
+keywords = ["topic modeling", "LLM", "multi-agent", "survey analysis", "data enrichment"]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence"
+]
+packages = [{include = "maltopic", from = "src"}]
+include = ["README.md", "LICENSE"]
+
+
+[tool.poetry.dependencies]
+python = "^3.12"
+openai = "^1.79.0"
+pandas = "^2.2.3"
+tqdm = "^4.67.1"
+
+
+[build-system]
+requires = ["poetry-core>=1.0.0"]
+build-backend = "poetry.core.masonry.api"

maltopic-1.0.0/src/maltopic/__init__.py

@@ -0,0 +1,16 @@
+# maltopic package initialization
+
+"""
+MALTopic: A Multi-Agent LLM Topic Modeling Library
+
+This package provides a framework for topic modeling using multiple LLM agents.
+Users can initialize the library with their API key and model name, and choose
+between different LLM providers (OpenAI, Google Gemini, Llama) for enhanced topic
+modeling capabilities.
+"""
+
+from .core import MALTopic
+
+__all__ = [
+    "MALTopic",
+]

maltopic-1.0.0/src/maltopic/core.py

@@ -0,0 +1,140 @@
+import json
+
+import pandas as pd
+from tqdm import tqdm
+
+from . import prompts, utils
+
+
+class MALTopic:
+    def __init__(self, api_key: str, default_model_name: str, llm_type: str):
+        self.api_key = api_key
+        self.default_model_name = default_model_name
+        self.llm_type = llm_type.lower()
+
+        self.llm_client = self._select_agent(api_key, default_model_name)
+
+    def _select_agent(self, api_key: str, model_name: str):
+        if self.llm_type == "openai":
+            from .llms import openai
+
+            return openai.OpenAIClient(api_key, model_name)
+        raise ValueError("Invalid LLM api type. Choose 'openai'.")
+
+    def enrich_free_text_with_structured_data(
+        self,
+        *,
+        survey_context: str,
+        free_text_column: str,
+        structured_data_columns: list[str],
+        df: pd.DataFrame,
+        examples: list[str] = [],
+    ) -> pd.DataFrame:
+        """
+        Enrich free text responses with structured data from other columns.
+
+        Args:
+            survey_context: Context about the survey to provide to the LLM
+            free_text_column: Name of the column containing free text responses
+            free_text_definition: Description of what the free text represents
+            structured_data_columns: List of column names containing structured data
+            df: Pandas DataFrame containing the data
+            examples: Optional list of example enrichments for few-shot learning
+
+        Returns:
+            DataFrame with added column '{free_text_column}_enriched' containing enriched text
+        """
+        utils.validate_dataframe(df, [free_text_column] + structured_data_columns)
+
+        instructions: str = prompts.ENRICH_INST.format(
+            survey_context=survey_context,
+            free_text_column=free_text_column,
+            structured_data_columns=", ".join(structured_data_columns),
+            examples="\n".join(examples) if examples else "No examples provided.",
+        )
+
+        enriched_column = f"{free_text_column}_enriched"
+        results: list[str] = []
+
+        for _, row in tqdm(df.iterrows(), total=len(df), desc="Enriching free text"):
+            free_text = str(row[free_text_column])
+            if not free_text or pd.isna(free_text):
+                results.append("")
+                continue
+
+            # Format structured data
+            structured_data: list[str] = []
+            for col in structured_data_columns:
+                if not pd.isna(row[col]):
+                    structured_data.append(f"{col}: {row[col]}")
+            structured_context: str = "\n".join(structured_data)
+
+            # Build complete prompt for this row
+            row_prompt: str = (
+                f"{free_text_column}: {free_text}\n\n"
+                f"Structured data:\n{structured_context}\n\n"
+                f"Enriched Response:"
+            )
+
+            try:
+                enriched_text: str = self.llm_client.generate(
+                    instructions=instructions, input=row_prompt
+                )
+                results.append(enriched_text.strip())
+            except Exception as e:
+                error_msg = f"Error generating enriched text: {str(e)}"
+                results.append(error_msg)
+
+        df[enriched_column] = results
+        return df
+
+    def generate_topics(
+        self, *, topic_mining_context: str, df: pd.DataFrame, enriched_column: str
+    ) -> list[dict[str, str]]:
+        """
+        Generate topics from enriched text responses.
+
+        Args:
+            topic_mining_context: Context about the survey to provide to the LLM. Add the what and why of the topic mining to make this useful.
+            df: Pandas DataFrame containing the data
+            enriched_column: Name of the column containing enriched text responses
+
+        Returns:
+            List of dictionaries, each representing a topic with 'id', 'name', and 'description'
+        """
+        utils.validate_dataframe(df, [enriched_column])
+
+        instructions = prompts.TOPIC_INST.format(survey_context=topic_mining_context)
+
+        all_columns: list[str] = df[enriched_column].dropna().tolist()
+        labeled_columns = [
+            f"{i+1}: {response}" for i, response in enumerate(all_columns)
+        ]
+        input_text = "\n\n".join(labeled_columns)
+
+        try:
+            raw_response = self.llm_client.generate(
+                instructions=instructions, input=input_text
+            )
+        except Exception as e:
+            raise RuntimeError(f"Error generating topics: {str(e)}")
+
+        topics = []
+
+        try:
+            parsed_topics = json.loads(raw_response)
+            for topic in parsed_topics:
+                for key in topic:
+                    if key != "representative_words" and not isinstance(
+                        topic[key], str
+                    ):
+                        topic[key] = str(topic[key])
+                topics.append(topic)
+        except json.JSONDecodeError:
+            raise ValueError(
+                f"Failed to parse LLM response as JSON: {raw_response[:100]}..."
+            )
+        except Exception as e:
+            raise ValueError(f"Error processing topics: {str(e)}")
+
+        return topics

maltopic-1.0.0/src/maltopic/llms/__init__.py

@@ -0,0 +1 @@
+# This file initializes the llms subpackage, allowing for structured imports from the LLM modules.

maltopic-1.0.0/src/maltopic/llms/openai.py

@@ -0,0 +1,32 @@
+from openai import OpenAI
+
+
+class OpenAIClient:
+    def __init__(self, api_key: str, model_name: str):
+        self.api_key = api_key
+        self.model_name = model_name
+        self.client = OpenAI(api_key=api_key)
+        self.temperature = 0.2
+        self.seed = 12345
+        self.top_p = 0.9
+
+    def generate(self, *, instructions: str, input: str) -> str:
+        response = self.client.chat.completions.create(
+            model=self.model_name,
+            store=False,
+            messages=[
+                {"role": "system", "content": instructions},
+                {"role": "user", "content": input},
+            ],
+            temperature=self.temperature,
+            top_p=self.top_p,
+            seed=self.seed,
+        )
+        if not response or not response.choices:
+            raise ValueError("No response received from OpenAI API.")
+        if not response.choices[0].message:
+            raise ValueError("No message in the response from OpenAI API.")
+        content = response.choices[0].message.content
+        if content is None:
+            raise ValueError("No content in the response message from OpenAI API.")
+        return content

maltopic-1.0.0/src/maltopic/prompts.py

@@ -0,0 +1,56 @@
+ENRICH_INST = (
+    "You are an AI language assistant. Your task is to help enrich the"
+    "free-text reponse with the structured data from survey responses."
+    "Enrich the free-text responses by summarizing them and injecting "
+    "crucial informationa and subtle details from the structured "
+    "columns. This should help with more context aware topic modeling "
+    "of the free-text responses. Survey context: {survey_context}. "
+    "Enrich the {free_text_column} with the following structured "
+    "columns: {structured_data_columns}. "
+    "Maintain the original sentiment and meaning of the response. Do "
+    "not introduce any new opinions, assumptions, conclusions or "
+    "extrapolations which were not present in the original response. "
+    "Keep the language generic and standardized. Only respond with "
+    "the enriched response. Some examples: {examples}."
+)
+
+TOPIC_INST = (
+    "You are an AI NLP data analyst. Your goal is to analyze a set of survey "
+    "responses and identify unique, exhaustive, and non-overlapping topics."
+    "Survey context: {survey_context}. "
+    "**Task:**"
+    "1. **Analyze the following survey responses.**  For each response, "
+    "note the respondent profile, the key themes and issues raised."
+    "2. **Identify Potential Topics.** Based on your analysis of all "
+    "responses, generate a list of initial, potentially granular topics. "
+    "Account for the distinct perspectives, experiences and background "
+    "different respondent might have."
+    "3. **Synthesize and Refine Topics.**  Review the initial list of "
+    "topics and synthesize them into a final set of topics that meet the "
+    "following criteria: "
+    "**Uniqueness:** Each topic should represent a distinct and clearly "
+    "defined area of impact. Avoid redundancy and overlapping concepts. "
+    "**Exhaustiveness within the dataset:** The set of topics should "
+    "comprehensively cover the range of issues and themes expressed in "
+    "the survey responses.  It should aim to capture all significant "
+    "aspects of the survey. "
+    "**Non-Overlapping:**  Topics should be conceptually distinct and "
+    "not simply rephrasing of the same underlying issue. Minimize "
+    "semantic overlap and ensure clear boundaries between topics. "
+    "**Respondent-Aware:** The topics should reflect the influence of "
+    "respondent profiles (if found). Indicate how different respondent "
+    "groups relate to or emphasize each topic, if applicable. "
+    "**Use all the data.** Use all the data provided to you. Do not "
+    "ignore any data. Do not make assumptions about the data. "
+    "4. **Output:**  Present your final output as a JSON array of topics. "
+    "For each topic, provide the following fields: "
+    "* `name`: A concise and descriptive name for the topic. "
+    "* `description`: A one line summary of what the topic encompasses. "
+    "* `relevance`: Note which respondent profiles are particularly "
+    "relevant to this topic, if applicable. Also include other interesting "
+    "and subtle patterns which you see related to this topic. "
+    "* `representative_words`: Array of top words which would represent "
+    "this topic in an NLP analysis."
+    "Only respond with the JSON array, with no additional text, "
+    "explanations or markdown."
+)

maltopic-1.0.0/src/maltopic/utils.py

@@ -0,0 +1,24 @@
+import pandas as pd
+
+
+def validate_dataframe(df: pd.DataFrame, required_columns: list[str]) -> None:
+    """
+    Validates a Pandas DataFrame for required columns and non-emptiness.
+
+    Args:
+        df: The Pandas DataFrame to validate.
+        required_columns: A list of column names that must be present in the DataFrame.
+
+    Raises:
+        ValueError: If the DataFrame is empty or if any required columns are missing.
+    """
+    if df.empty:
+        raise ValueError("DataFrame is empty. It must contain at least one row.")
+
+    missing_columns = [col for col in required_columns if col not in df.columns]
+    if missing_columns:
+        raise ValueError(
+            f"DataFrame is missing required columns: {', '.join(missing_columns)}"
+        )
+
+    return None