multi-llm-consensus 0.1.3__tar.gz

multi_llm_consensus-0.1.3/.claude/settings.local.json

@@ -0,0 +1,22 @@
+{
+  "permissions": {
+    "allow": [
+      "WebSearch",
+      "Bash(uv add:*)",
+      "Bash(tree:*)",
+      "Bash(uv run:*)",
+      "WebFetch(domain:docs.langchain.com)",
+      "WebFetch(domain:smith.langchain.com)",
+      "Bash(jq:*)",
+      "Bash(python:*)",
+      "Bash(python -c:*)",
+      "WebFetch(domain:github.com)",
+      "WebFetch(domain:raw.githubusercontent.com)",
+      "Bash(.venv/bin/python:*)",
+      "Bash(uv build:*)",
+      "Bash(unzip:*)",
+      "Bash(uv publish:*)",
+      "Bash(find:*)"
+    ]
+  }
+}

multi_llm_consensus-0.1.3/.gitignore

@@ -0,0 +1,14 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+
+# Virtual environments
+.venv
+
+# Environment variables
+.env
+debug/

multi_llm_consensus-0.1.3/.python-version

@@ -0,0 +1 @@
+3.12

multi_llm_consensus-0.1.3/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Roberto Pagliari
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

multi_llm_consensus-0.1.3/PKG-INFO

@@ -0,0 +1,171 @@
+Metadata-Version: 2.4
+Name: multi-llm-consensus
+Version: 0.1.3
+Summary: Achieve consensus across multiple LLMs through an AI judge coordinator
+Project-URL: Homepage, https://github.com/zzzrbx/llm-ensemble
+Project-URL: Repository, https://github.com/zzzrbx/llm-ensemble
+Project-URL: Issues, https://github.com/zzzrbx/llm-ensemble/issues
+Author-email: Roberto Pagliari <roberto.pagliari@gmail.com>
+License: MIT
+License-File: LICENSE
+Keywords: agent,ai,consensus,langchain,langgraph,llm,multi-model
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.12
+Requires-Dist: deepagents>=0.1.0
+Requires-Dist: langchain-anthropic>=1.3.1
+Requires-Dist: langchain-community>=0.4.1
+Requires-Dist: langchain-google-genai>=4.1.3
+Requires-Dist: langchain-openai>=1.1.7
+Requires-Dist: langchain-xai>=1.2.1
+Requires-Dist: langchain>=1.2.3
+Requires-Dist: langgraph>=1.0.5
+Requires-Dist: pydantic>=2.12.5
+Requires-Dist: python-dotenv>=1.2.1
+Requires-Dist: rich>=14.2.0
+Requires-Dist: tavily-python>=0.7.17
+Description-Content-Type: text/markdown
+
+# LLM Ensemble
+
+![LLM Ensemble Banner](.github/banner.png)
+
+A Python library for achieving consensus across multiple Agents.
+
+## Features
+
+- **Consensus**: Uses a moderator to iteratively coordinate multiple LLMs until consensus is reached
+- Can use any model API based model such as OpenAI, Anthropic, Gemini, Grok supported by Langgraph
+- Supports web search for real-time data 
+
+## Installation
+
+```bash
+uv add multi-llm-consensus
+```
+
+Or install from source:
+
+```bash
+git clone https://github.com/zzzrbx/llm-ensemble.git
+cd llm-ensemble
+uv sync
+```
+
+## Environment Setup
+
+Create a `.env` file with your API keys, for example:
+
+```bash
+OPENAI_API_KEY=your_openai_key
+ANTHROPIC_API_KEY=your_anthropic_key
+GOOGLE_API_KEY=your_google_key
+XAI_API_KEY=your_xai_key
+TAVILY_API_KEY=your_tavily_key  # For web search
+```
+
+You must provide at least two API keys for the models you want to use in the ensemble.
+
+## How It Works
+
+```
+User Query → Judge (configurable, default: Claude Opus 4.5)
+    ↓
+Judge calls run_llms tool
+    ├── Model A (parallel)
+    ├── Model B (parallel)
+    ├── Model C (parallel)
+    └── Model D (parallel)
+         ↓
+Judge analyzes responses
+    ├── Consensus? → Return answer
+    └── No consensus? → Refine query and call run_llms again
+         ↓
+Repeat until consensus or limit reached
+```
+
+**Key Features:**
+- **Dynamic queries** - Judge crafts different prompts each iteration:
+  - Iteration 1: Sends initial question with research instructions
+  - Iteration 2+: Summarizes agreements, highlights disagreements, requests refinements
+  - Final iteration: Presents refined consensus statement for confirmation
+- **Error handling** - Returns default values on timeout or tool call limit reached
+
+**Tools currently available for LLMs:**
+- `search_the_web` - Tavily web search for current events and factual data
+- `add`, `subtract`, `multiply`, `divide` - Math operations
+
+
+## Examples
+
+### Example 1: With structured output
+
+```python
+from typing import TypedDict
+from llm_ensemble import Consensus
+
+class UserSchema(TypedDict):
+    consensus: bool
+    final_answer: str
+    notes: str
+
+consensus = Consensus(
+    models=[
+        "openai:gpt-5-mini",
+        "google_genai:gemini-3-flash-preview",
+        "anthropic:claude-3-5-haiku-20241022",
+        "xai:grok-3-mini",
+    ],
+    response_schema=UserSchema
+)
+
+result = consensus.invoke(
+    "If survival is arbitrary, is moral judgment arbitrary too?"
+)
+
+print(f"Consensus: {result['consensus']}")
+print(f"Answer: {result['final_answer']}")
+print(f"Notes: {result['notes']}")
+```
+
+### Example 2: No structured output with web search enabled (you just need to mention it in the prompt)
+
+```python
+from llm_ensemble import Consensus
+
+# No response_schema - returns full agent result
+consensus = Consensus(
+    models=[
+        "openai:gpt-5-mini",
+        "google_genai:gemini-3-flash-preview",
+        "anthropic:claude-3-5-haiku-20241022",
+        "xai:grok-3-mini",
+    ]
+)
+
+result = consensus.invoke(
+    "What are the latest developments in quantum computing?\n\n"
+    "Use the web search to research current news and breakthroughs."
+)
+
+# Access full agent result
+print(result['messages'][-1].content)
+```
+
+## Debugging and Observability
+
+The library integrates with LangSmith for trace observability. Set `LANGSMITH_API_KEY` and `LANGSMITH_PROJECT` in your `.env` file to enable tracing.
+
+## License
+
+MIT License
+
+## Contributing
+
+Contributions are welcome! Please open an issue or submit a pull request.
+
+

multi_llm_consensus-0.1.3/README.md

@@ -0,0 +1,139 @@
+# LLM Ensemble
+
+![LLM Ensemble Banner](.github/banner.png)
+
+A Python library for achieving consensus across multiple Agents.
+
+## Features
+
+- **Consensus**: Uses a moderator to iteratively coordinate multiple LLMs until consensus is reached
+- Can use any model API based model such as OpenAI, Anthropic, Gemini, Grok supported by Langgraph
+- Supports web search for real-time data 
+
+## Installation
+
+```bash
+uv add multi-llm-consensus
+```
+
+Or install from source:
+
+```bash
+git clone https://github.com/zzzrbx/llm-ensemble.git
+cd llm-ensemble
+uv sync
+```
+
+## Environment Setup
+
+Create a `.env` file with your API keys, for example:
+
+```bash
+OPENAI_API_KEY=your_openai_key
+ANTHROPIC_API_KEY=your_anthropic_key
+GOOGLE_API_KEY=your_google_key
+XAI_API_KEY=your_xai_key
+TAVILY_API_KEY=your_tavily_key  # For web search
+```
+
+You must provide at least two API keys for the models you want to use in the ensemble.
+
+## How It Works
+
+```
+User Query → Judge (configurable, default: Claude Opus 4.5)
+    ↓
+Judge calls run_llms tool
+    ├── Model A (parallel)
+    ├── Model B (parallel)
+    ├── Model C (parallel)
+    └── Model D (parallel)
+         ↓
+Judge analyzes responses
+    ├── Consensus? → Return answer
+    └── No consensus? → Refine query and call run_llms again
+         ↓
+Repeat until consensus or limit reached
+```
+
+**Key Features:**
+- **Dynamic queries** - Judge crafts different prompts each iteration:
+  - Iteration 1: Sends initial question with research instructions
+  - Iteration 2+: Summarizes agreements, highlights disagreements, requests refinements
+  - Final iteration: Presents refined consensus statement for confirmation
+- **Error handling** - Returns default values on timeout or tool call limit reached
+
+**Tools currently available for LLMs:**
+- `search_the_web` - Tavily web search for current events and factual data
+- `add`, `subtract`, `multiply`, `divide` - Math operations
+
+
+## Examples
+
+### Example 1: With structured output
+
+```python
+from typing import TypedDict
+from llm_ensemble import Consensus
+
+class UserSchema(TypedDict):
+    consensus: bool
+    final_answer: str
+    notes: str
+
+consensus = Consensus(
+    models=[
+        "openai:gpt-5-mini",
+        "google_genai:gemini-3-flash-preview",
+        "anthropic:claude-3-5-haiku-20241022",
+        "xai:grok-3-mini",
+    ],
+    response_schema=UserSchema
+)
+
+result = consensus.invoke(
+    "If survival is arbitrary, is moral judgment arbitrary too?"
+)
+
+print(f"Consensus: {result['consensus']}")
+print(f"Answer: {result['final_answer']}")
+print(f"Notes: {result['notes']}")
+```
+
+### Example 2: No structured output with web search enabled (you just need to mention it in the prompt)
+
+```python
+from llm_ensemble import Consensus
+
+# No response_schema - returns full agent result
+consensus = Consensus(
+    models=[
+        "openai:gpt-5-mini",
+        "google_genai:gemini-3-flash-preview",
+        "anthropic:claude-3-5-haiku-20241022",
+        "xai:grok-3-mini",
+    ]
+)
+
+result = consensus.invoke(
+    "What are the latest developments in quantum computing?\n\n"
+    "Use the web search to research current news and breakthroughs."
+)
+
+# Access full agent result
+print(result['messages'][-1].content)
+```
+
+## Debugging and Observability
+
+The library integrates with LangSmith for trace observability. Set `LANGSMITH_API_KEY` and `LANGSMITH_PROJECT` in your `.env` file to enable tracing.
+
+## License
+
+MIT License
+
+## Contributing
+
+Contributions are welcome! Please open an issue or submit a pull request.
+
+

multi_llm_consensus-0.1.3/llm_ensemble/__init__.py

@@ -0,0 +1,4 @@
+from .run_llm import RunLLM
+from .consensus import Consensus
+
+__all__ = ["RunLLM", "Consensus"]

multi_llm_consensus-0.1.3/llm_ensemble/consensus.py

@@ -0,0 +1,165 @@
+from langchain.agents import create_agent
+from langchain.agents.middleware import TodoListMiddleware, SummarizationMiddleware, ToolCallLimitMiddleware
+from deepagents.middleware.filesystem import FilesystemMiddleware
+from deepagents.backends import StateBackend
+from langchain.chat_models import init_chat_model
+from langchain.tools import tool
+from .run_llm import RunLLM
+from pathlib import Path
+from typing import Type
+
+
+class Consensus:
+    """
+    Consensus class that uses a configurable judge model to orchestrate
+    multiple RunLLM invocations until consensus is reached among LLMs.
+    """
+
+    def __init__(
+        self,
+        models: list[str],
+        judge_model: str = "anthropic:claude-opus-4-5-20251101",
+        summarization_model: str = "claude-4-5-sonnet-20250929",
+        summarization_trigger_tokens: int = 200_000,
+        summarization_keep_messages: int = 5,
+        run_limit: int = 20,
+        response_schema: Type | None = None
+    ):
+        """
+        Initialize the Consensus class.
+
+        Args:
+            models: List of model strings in format "provider:model-name".
+            judge_model: Model string for the judge coordinator in format "provider:model-name".
+                        Defaults to "anthropic:claude-opus-4-5-20251101".
+            summarization_model: Model string for summarization middleware in format "provider:model-name".
+                        Defaults to "anthropic:claude-3-5-sonnet-20241022".
+            summarization_trigger_tokens: Token count to trigger summarization middleware.
+            summarization_keep_messages: Number of messages to keep after summarization.
+            run_limit: Maximum number of calls to run_llms tool per invocation.
+            response_schema: Optional schema for structured output (TypedDict or Pydantic model).
+                            If None, returns full agent result without structured output.
+
+        Raises:
+            ValueError: If models list is empty or contains only one model.
+        """
+        if not models:
+            raise ValueError("models list cannot be empty")
+        if len(models) < 2:
+            raise ValueError("models list must contain at least 2 models for consensus")
+
+        # Store as instance variables for use in tool creation
+        self.models = models
+        self.system_message = "You are a helpful AI assistant."
+
+        # Load judge prompt
+        judge_prompt_path = Path(__file__).parent / "prompts" / "judge.prompt"
+        judge_prompt = judge_prompt_path.read_text()
+
+        # Create the run_llms tool
+        run_llms = self._create_run_llms_tool()
+
+        # Create judge LLM using init_chat_model
+        llm = init_chat_model(judge_model)
+
+        # Create middleware
+        middleware = [
+            TodoListMiddleware(),
+            FilesystemMiddleware(backend=lambda rt: StateBackend(rt)),
+            SummarizationMiddleware(
+                model=summarization_model,
+                trigger=("tokens", summarization_trigger_tokens),
+                keep=("messages", summarization_keep_messages)
+            ),
+            ToolCallLimitMiddleware(
+                tool_name="run_llms",
+                run_limit=run_limit,
+                exit_behavior="error"
+            )
+        ]
+
+        # Create agent with optional structured output
+        if response_schema is not None:
+            self._agent = create_agent(
+                model=llm,
+                tools=[run_llms],
+                system_prompt=judge_prompt,
+                middleware=middleware,
+                response_format=response_schema
+            )
+        else:
+            self._agent = create_agent(
+                model=llm,
+                tools=[run_llms],
+                system_prompt=judge_prompt,
+                middleware=middleware
+            )
+
+        self._response_schema = response_schema
+
+    def _create_run_llms_tool(self):
+        """
+        Creates the run_llms tool with access to instance variables.
+
+        Returns:
+            A LangChain tool that runs multiple LLMs in parallel.
+        """
+        @tool
+        def run_llms(query: str) -> str:
+            """
+            Runs multiple LLMs in parallel on the same query.
+
+            Args:
+                query: The prompt/question to send to all LLMs. Include full context and
+                       any specific instructions (e.g., "use search_the_web for web search",
+                       "use add/multiply/subtract/divide tools for calculations").
+
+            Returns:
+                Aggregated responses from all LLMs. Each response is prefixed with the
+                exact model identifier (e.g., "openai:gpt-5-mini:", "google_genai:gemini-3-flash-preview:").
+                Always refer to models by these exact identifiers in your analysis.
+            """
+            run_llm = RunLLM(models=self.models, system_message=self.system_message)
+            return run_llm.invoke(query)
+
+        return run_llms
+
+    def invoke(self, prompt: str):
+        """
+        Invoke the consensus process with a user query.
+
+        Args:
+            prompt: The user's initial query
+
+        Returns:
+            If response_schema was provided: structured response dict (or default values on error)
+            Otherwise: full agent result (or None on error)
+        """
+        try:
+            result = self._agent.invoke({
+                "messages": [{"role": "user", "content": prompt}]
+            })
+            if self._response_schema is not None:
+                return result["structured_response"]
+            else:
+                return result
+        except Exception as e:
+            # Tool call limit reached or other error
+            print(f"Error during consensus: {str(e)}")
+            if self._response_schema is not None:
+                # Return default values - create a dict with all schema keys set to defaults
+                # This tries to match common schema patterns
+                default_dict = {}
+                if hasattr(self._response_schema, '__annotations__'):
+                    for key, type_hint in self._response_schema.__annotations__.items():
+                        if key in ['consensus', 'consensus_reached']:
+                            default_dict[key] = False
+                        elif type_hint == bool:
+                            default_dict[key] = False
+                        elif type_hint == str:
+                            default_dict[key] = f"Error occurred: {str(e)}"
+                        else:
+                            default_dict[key] = None
+                return default_dict if default_dict else None
+            else:
+                return None

multi_llm_consensus-0.1.3/llm_ensemble/prompts/judge.prompt

@@ -0,0 +1,38 @@
+You are a consensus coordinator for multiple AI language models. Your role is to facilitate agreement among different LLMs by iteratively querying them and refining your questions based on their responses.
+
+## Your Tools (Judge)
+
+**run_llms**: Runs multiple LLMs in parallel. Takes a `query` parameter where you craft the prompt to send to all LLMs.
+
+**TODO list**: Track agreements reached, disagreements to resolve, and clarifications needed across iterations.
+
+**Filesystem** (`write_file`, `read_file`, `ls`): After the first run_llms call, save the exact LLM model identifiers to a file and add a TODO reminder to read this file before providing the final answer to ensure model names are accurate.
+
+## Your Process
+
+1. **Initial Query**: Call `run_llms` with the user's question
+2. **Analyze Responses**: Identify agreements, disagreements, and areas needing clarification
+3. **Iterative Refinement**: Call `run_llms` again with updated queries that summarize the objective/context and clarify specific points where models diverged
+4. **Continue Until Consensus**: Iterate until LLMs substantially agree or you can draw a reasonable conclusion
+
+## Context Management - CRITICAL
+
+**LLMs ARE STATELESS.** They have NO memory of previous iterations. In every query after the first, you MUST include:
+
+1. **Original Question**: Restate the user's question or provide a summary
+2. **Previous Iterations**: Summarize what each model said in prior rounds. **IMPORTANT: Always refer to models by their exact identifiers as shown in the run_llms output** (e.g., "openai:gpt-5-mini said X", "google_genai:gemini-3-flash-preview said Y"). Never use shortened or friendly names.
+3. **Current Status**: Explain where you are now, what's agreed upon, and what needs resolution
+4. **Complete Context**: Treat each query as if the LLMs are starting fresh—provide everything they need to answer meaningfully
+
+## LLM Tools (Instruct LLMs to Use)
+
+When crafting queries for `run_llms`, instruct the LLMs to use their available tools:
+
+- **search_the_web**: For current events, recent information, factual verification, or when the user requests web search. If used in iteration 1, in subsequent iterations tell LLMs to use only if needed
+- **Calculation tools** (`add`, `subtract`, `multiply`, `divide`): If the user query implies calculations, tell LLMs to use these tools
+
+## Critical Guidelines
+
+- **Use exact model identifiers**: Always refer to models by their full identifiers as they appear in run_llms 
+- Base consensus ONLY on what the LLMs tell you, not your own knowledge
+- Be completely unbiased—evaluate all responses objectively and equally