agentic-qa 0.1.0__tar.gz

agentic_qa-0.1.0/PKG-INFO

@@ -0,0 +1,180 @@
+Metadata-Version: 2.4
+Name: agentic_qa
+Version: 0.1.0
+Summary: Autonomous Agentic QA System for testing RAG pipelines and LLM systems.
+Home-page: https://github.com/yourusername/multi-agent-qa
+Author: Your Name
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: langgraph>=0.2.0
+Requires-Dist: langchain>=0.3.0
+Requires-Dist: langchain-openai>=0.2.0
+Requires-Dist: langchain-core>=0.3.0
+Requires-Dist: langsmith>=0.1.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: requests>=2.30.0
+Requires-Dist: ragas>=0.1.0
+Requires-Dist: datasets>=2.0.0
+Dynamic: author
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# 🛡️ Agentic QA: Autonomous Multi-Agent Testing for RAG & LLMs
+
+**Agentic QA** is a Python library that autonomously generates adversarial test cases, executes them against your RAG/LLM system, evaluates the results, and self-improves its testing coverage—all without human intervention.
+
+Unlike traditional testing frameworks (like RAGAS or TruLens) that evaluate outputs against static, human-written inputs, **Agentic QA acts as an active red-team**, dynamically generating the tricky edge cases needed to break your system.
+
+![Python](https://img.shields.io/badge/Python-3.10+-3776AB?logo=python&logoColor=white)
+![LangGraph](https://img.shields.io/badge/LangGraph-0.2+-1C3C3C?logo=langchain&logoColor=white)
+![LangSmith](https://img.shields.io/badge/LangSmith-Monitored-FF6B35?logo=langchain&logoColor=white)
+![Streamlit](https://img.shields.io/badge/Streamlit-Dashboard-FF4B4B?logo=streamlit&logoColor=white)
+
+---
+
+## 🚀 Quick Start
+
+### 1. Installation
+
+Install the library locally:
+
+```bash
+git clone https://github.com/yourusername/multi-agent-qa.git
+cd multi-agent-qa
+pip install -e .
+```
+
+Ensure you have your `.env` configured with your API keys:
+
+```bash
+cp .env.example .env
+# Edit .env and provide OPENAI_API_KEY
+```
+
+### 2. Using the Python Library
+
+You can test any RAG or LLM pipeline in just a few lines of code.
+
+#### Option A: Testing a Python Function
+If your RAG system is a Python function in your codebase:
+
+```python
+import agentic_qa
+
+# Your existing RAG or Chatbot function
+def my_custom_rag(query: str) -> str:
+    # Example: return my_langchain_pipeline.invoke(query)
+    return "This is my AI response."
+
+# Run the autonomous testing loop
+report = agentic_qa.run_autonomous_test(
+    target_function=my_custom_rag,
+    system_name="YouTube Video Q&A",
+    system_description="A chatbot that answers questions about YouTube transcripts.",
+    domain="video content",
+    max_iterations=3,          # How many times agents learn and retry
+    tests_per_iteration=5      # Tests generated per round
+)
+```
+
+#### Option B: Testing an API Endpoint
+If your system is deployed behind a REST API (FastAPI, Flask, LangServe):
+
+```python
+import agentic_qa
+
+report = agentic_qa.run_autonomous_test(
+    api_endpoint="http://localhost:8000/api/chat",
+    system_name="Customer Support Bot",
+    system_description="An AI that resolves customer support tickets.",
+    domain="customer support"
+)
+```
+
+### 3. Using the Streamlit UI
+
+If you prefer a visual dashboard to monitor the agents in real-time, run the included Streamlit app:
+
+```bash
+streamlit run app.py
+```
+
+From the UI, you can connect your API endpoint or use the built-in mock system for a demonstration.
+
+---
+
+## 🏗️ Architecture
+
+The framework is powered by 5 autonomous agents built with LangGraph:
+
+```
+START ──▶ 🔴 Red-Team Agent ──▶ ⚡ Executor Agent ──▶ ⚖️ Judge Agent ──▶ Decision
+              ▲                                                            │
+              │                                                      ┌─────┴─────┐
+              │                                                      ▼           ▼
+              └──────────────── 🔧 Refiner Agent               📊 Reporter Agent
+                                   (loop back)                       (END)
+```
+
+### Agent Roles
+
+| Agent | Role |
+|-------|------|
+| 🔴 **Red-Team** | Generates adversarial test inputs targeting edge cases (prompt injections, boundary values, etc.). |
+| ⚡ **Executor** | Runs tests through the target system and captures the outputs. |
+| ⚖️ **Judge** | Evaluates the outputs using an LLM-as-a-Judge pattern with strict pass/fail criteria. |
+| 🔧 **Refiner** | Analyzes the judge's failure patterns and instructs the Red-Team on how to exploit weaknesses in the next iteration. |
+| 📊 **Reporter** | Compiles a comprehensive final Markdown QA report. |
+
+---
+
+## 🧠 What Makes This Novel
+
+| Traditional Testing Tools (RAGAS, TruLens) | Agentic QA |
+|----------------------------------------|-------------|
+| Measures outputs against static inputs | **Generates** the adversarial inputs autonomously |
+| Human writes test cases | AI agents write and refine test cases |
+| One-shot evaluation | **Self-improving loop** with pattern learning |
+| Relies heavily on reference data | Relies on behavioral boundaries and edge-case testing |
+
+---
+
+## 📂 Project Structure
+
+```text
+multi-agent-qa/
+├── agentic_qa/
+│   ├── __init__.py           # Clean developer API (run_autonomous_test)
+│   ├── agents/               # 5 LangGraph agent definitions
+│   ├── graph/                # State definitions and LangGraph flow
+│   ├── schemas/              # Pydantic validation models
+│   ├── sut/                  # Adapters (API, Callable, Base)
+│   └── utils/                # Prompt templates
+├── setup.py                  # Package configuration
+├── app.py                    # Streamlit Dashboard UI
+├── .env                      # API Keys configuration
+└── README.md
+```
+
+## 📡 LangSmith Monitoring
+
+All agent interactions are automatically traced via LangSmith if configured in `.env`.
+
+```env
+LANGCHAIN_TRACING_V2=true
+LANGCHAIN_API_KEY=your-langsmith-api-key
+LANGCHAIN_PROJECT=agentic-qa
+```
+
+## 📄 License
+
+MIT

agentic_qa-0.1.0/README.md

@@ -0,0 +1,150 @@
+# 🛡️ Agentic QA: Autonomous Multi-Agent Testing for RAG & LLMs
+
+**Agentic QA** is a Python library that autonomously generates adversarial test cases, executes them against your RAG/LLM system, evaluates the results, and self-improves its testing coverage—all without human intervention.
+
+Unlike traditional testing frameworks (like RAGAS or TruLens) that evaluate outputs against static, human-written inputs, **Agentic QA acts as an active red-team**, dynamically generating the tricky edge cases needed to break your system.
+
+![Python](https://img.shields.io/badge/Python-3.10+-3776AB?logo=python&logoColor=white)
+![LangGraph](https://img.shields.io/badge/LangGraph-0.2+-1C3C3C?logo=langchain&logoColor=white)
+![LangSmith](https://img.shields.io/badge/LangSmith-Monitored-FF6B35?logo=langchain&logoColor=white)
+![Streamlit](https://img.shields.io/badge/Streamlit-Dashboard-FF4B4B?logo=streamlit&logoColor=white)
+
+---
+
+## 🚀 Quick Start
+
+### 1. Installation
+
+Install the library locally:
+
+```bash
+git clone https://github.com/yourusername/multi-agent-qa.git
+cd multi-agent-qa
+pip install -e .
+```
+
+Ensure you have your `.env` configured with your API keys:
+
+```bash
+cp .env.example .env
+# Edit .env and provide OPENAI_API_KEY
+```
+
+### 2. Using the Python Library
+
+You can test any RAG or LLM pipeline in just a few lines of code.
+
+#### Option A: Testing a Python Function
+If your RAG system is a Python function in your codebase:
+
+```python
+import agentic_qa
+
+# Your existing RAG or Chatbot function
+def my_custom_rag(query: str) -> str:
+    # Example: return my_langchain_pipeline.invoke(query)
+    return "This is my AI response."
+
+# Run the autonomous testing loop
+report = agentic_qa.run_autonomous_test(
+    target_function=my_custom_rag,
+    system_name="YouTube Video Q&A",
+    system_description="A chatbot that answers questions about YouTube transcripts.",
+    domain="video content",
+    max_iterations=3,          # How many times agents learn and retry
+    tests_per_iteration=5      # Tests generated per round
+)
+```
+
+#### Option B: Testing an API Endpoint
+If your system is deployed behind a REST API (FastAPI, Flask, LangServe):
+
+```python
+import agentic_qa
+
+report = agentic_qa.run_autonomous_test(
+    api_endpoint="http://localhost:8000/api/chat",
+    system_name="Customer Support Bot",
+    system_description="An AI that resolves customer support tickets.",
+    domain="customer support"
+)
+```
+
+### 3. Using the Streamlit UI
+
+If you prefer a visual dashboard to monitor the agents in real-time, run the included Streamlit app:
+
+```bash
+streamlit run app.py
+```
+
+From the UI, you can connect your API endpoint or use the built-in mock system for a demonstration.
+
+---
+
+## 🏗️ Architecture
+
+The framework is powered by 5 autonomous agents built with LangGraph:
+
+```
+START ──▶ 🔴 Red-Team Agent ──▶ ⚡ Executor Agent ──▶ ⚖️ Judge Agent ──▶ Decision
+              ▲                                                            │
+              │                                                      ┌─────┴─────┐
+              │                                                      ▼           ▼
+              └──────────────── 🔧 Refiner Agent               📊 Reporter Agent
+                                   (loop back)                       (END)
+```
+
+### Agent Roles
+
+| Agent | Role |
+|-------|------|
+| 🔴 **Red-Team** | Generates adversarial test inputs targeting edge cases (prompt injections, boundary values, etc.). |
+| ⚡ **Executor** | Runs tests through the target system and captures the outputs. |
+| ⚖️ **Judge** | Evaluates the outputs using an LLM-as-a-Judge pattern with strict pass/fail criteria. |
+| 🔧 **Refiner** | Analyzes the judge's failure patterns and instructs the Red-Team on how to exploit weaknesses in the next iteration. |
+| 📊 **Reporter** | Compiles a comprehensive final Markdown QA report. |
+
+---
+
+## 🧠 What Makes This Novel
+
+| Traditional Testing Tools (RAGAS, TruLens) | Agentic QA |
+|----------------------------------------|-------------|
+| Measures outputs against static inputs | **Generates** the adversarial inputs autonomously |
+| Human writes test cases | AI agents write and refine test cases |
+| One-shot evaluation | **Self-improving loop** with pattern learning |
+| Relies heavily on reference data | Relies on behavioral boundaries and edge-case testing |
+
+---
+
+## 📂 Project Structure
+
+```text
+multi-agent-qa/
+├── agentic_qa/
+│   ├── __init__.py           # Clean developer API (run_autonomous_test)
+│   ├── agents/               # 5 LangGraph agent definitions
+│   ├── graph/                # State definitions and LangGraph flow
+│   ├── schemas/              # Pydantic validation models
+│   ├── sut/                  # Adapters (API, Callable, Base)
+│   └── utils/                # Prompt templates
+├── setup.py                  # Package configuration
+├── app.py                    # Streamlit Dashboard UI
+├── .env                      # API Keys configuration
+└── README.md
+```
+
+## 📡 LangSmith Monitoring
+
+All agent interactions are automatically traced via LangSmith if configured in `.env`.
+
+```env
+LANGCHAIN_TRACING_V2=true
+LANGCHAIN_API_KEY=your-langsmith-api-key
+LANGCHAIN_PROJECT=agentic-qa
+```
+
+## 📄 License
+
+MIT

agentic_qa-0.1.0/agentic_qa/__init__.py

@@ -0,0 +1,103 @@
+"""
+Agentic QA - Main Public API
+
+This file exposes the simple `run_autonomous_test` function, allowing developers 
+to test their RAG systems in just a few lines of code.
+"""
+
+import os
+from typing import Callable, Optional
+
+# Expose SUT adapters for developers if they want advanced setups
+from agentic_qa.sut.base import BaseSUTAdapter
+from agentic_qa.sut.api_adapter import APIAdapter
+from agentic_qa.sut.callable_adapter import CallableAdapter
+from agentic_qa.sut import set_active_sut
+
+# Import the core workflow
+from agentic_qa.graph.workflow import build_qa_graph, get_initial_state
+
+
+def run_autonomous_test(
+    target_function: Optional[Callable] = None,
+    api_endpoint: Optional[str] = None,
+    system_name: str = "Target System",
+    system_description: str = "A generic RAG system",
+    domain: str = "general",
+    max_iterations: int = 3,
+    tests_per_iteration: int = 5,
+    model_name: str = "gpt-4o-mini",
+) -> dict:
+    """
+    Run an autonomous multi-agent QA test against a target system.
+    
+    You must provide EITHER a `target_function` (a python function) OR 
+    an `api_endpoint` (a URL string).
+    
+    Args:
+        target_function: A python function that takes a string query and returns a string answer.
+        api_endpoint: A URL endpoint (e.g., http://localhost:8000/chat) to test.
+        system_name: The name of the system being tested.
+        system_description: A description of what the system does. Highly important for agents!
+        domain: The domain of the system (e.g., 'financial', 'healthcare', 'customer support').
+        max_iterations: How many times the agents should refine and retry their tests.
+        tests_per_iteration: How many tests the Red-Team agent generates per round.
+        model_name: The LLM to use for the agents (default: gpt-4o-mini).
+        
+    Returns:
+        A dictionary containing the final execution state, including the test suite, verdicts,
+        failure patterns, and the final Markdown report.
+    """
+    # 1. Configure Environment variables needed by LangGraph
+    os.environ["MAX_ITERATIONS"] = str(max_iterations)
+    os.environ["TESTS_PER_ITERATION"] = str(tests_per_iteration)
+    os.environ["MODEL_NAME"] = model_name
+    
+    # 2. Setup the SUT Adapter
+    if target_function:
+        adapter = CallableAdapter(
+            fn=target_function,
+            description=system_description,
+            system_name=system_name,
+            domain=domain
+        )
+    elif api_endpoint:
+        adapter = APIAdapter(
+            endpoint=api_endpoint,
+            description=system_description,
+            system_name=system_name,
+            domain=domain
+        )
+    else:
+        raise ValueError("You must provide either a `target_function` or an `api_endpoint`.")
+
+    # Register the adapter globally for the Executor Agent to use
+    set_active_sut(adapter)
+    
+    # 3. Build and Run the Graph
+    print(f"🚀 Starting Autonomous QA Test against: {system_name}")
+    print(f"Domain: {domain} | Max Iterations: {max_iterations}")
+    
+    graph = build_qa_graph()
+    initial_state = get_initial_state()
+    initial_state["max_iterations"] = max_iterations
+    
+    final_state = None
+    # Stream the graph to provide real-time console feedback
+    for event in graph.stream(initial_state, stream_mode="values"):
+        final_state = event
+        
+    print("\n✅ Autonomous QA Test Complete!")
+    print(f"Coverage Score: {final_state.get('coverage_score', 0):.1%}")
+    print(f"Total Failure Patterns Found: {len(final_state.get('failure_patterns', []))}")
+    
+    return final_state
+
+
+# Define what is exported when someone runs `from agentic_qa import *`
+__all__ = [
+    "run_autonomous_test",
+    "APIAdapter",
+    "CallableAdapter",
+    "BaseSUTAdapter"
+]

agentic_qa-0.1.0/agentic_qa/agents/__init__.py

@@ -0,0 +1,15 @@
+"""Multi-Agent Autonomous QA System - Agents Package"""
+
+from agentic_qa.agents.red_team import red_team_node
+from agentic_qa.agents.executor import executor_node
+from agentic_qa.agents.judge import judge_node
+from agentic_qa.agents.refiner import refiner_node
+from agentic_qa.agents.reporter import reporter_node
+
+__all__ = [
+    "red_team_node",
+    "executor_node", 
+    "judge_node",
+    "refiner_node",
+    "reporter_node",
+]

agentic_qa-0.1.0/agentic_qa/agents/discovery.py

@@ -0,0 +1,88 @@
+"""
+Discovery Agent (Graphify) — Architecture Mapping.
+
+This agent performs White-Box introspection. It analyzes the System Under Test (SUT)
+and maps out its internal architecture (e.g., Vector DB, Chunk Size, Retriever type, LLM).
+This architectural "graph" allows the Red-Team agent to launch hyper-targeted attacks.
+"""
+
+import os
+from langchain_openai import ChatOpenAI
+from langchain_core.messages import SystemMessage, HumanMessage
+from agentic_qa.graph.state import QAState
+
+DISCOVERY_SYSTEM_PROMPT = """You are an elite AI Architecture Mapper (Graphify Agent).
+
+Your mission is to analyze a generic description of an AI/RAG system and deduce its likely internal architecture graph.
+You must break down the system into a logical pipeline (e.g., Data Ingestion -> Chunking -> VectorDB -> Retriever -> LLM -> Output).
+
+Think about:
+1. What components must exist for this system to work?
+2. Where are the likely weak points or bottlenecks between these nodes?
+3. What are the assumed configurations (e.g., chunk size, top-k retrieval)?
+
+Output a detailed, graph-like text representation of the architecture. Be specific about potential vulnerabilities at each node."""
+
+DISCOVERY_USER_PROMPT = """Analyze the following System Under Test and map its architecture.
+
+**System Name:** {name}
+**Domain:** {domain}
+**Description:**
+{description}
+
+Provide a detailed architectural breakdown (Graphify) of this system. Highlight the specific nodes (e.g., Retriever, VectorStore, LLM) and the data flow. 
+Keep it concise but technical."""
+
+
+def _get_llm() -> ChatOpenAI:
+    return ChatOpenAI(
+        model=os.getenv("MODEL_NAME", "gpt-4o-mini"),
+        temperature=0.2,
+        openai_api_key=os.getenv("OPENAI_API_KEY"),
+        openai_api_base=os.getenv("OPENAI_API_BASE", "https://api.openai.com/v1"),
+    )
+
+
+def discovery_node(state: QAState) -> dict:
+    """
+    LangGraph node: Discovery Agent.
+    
+    Runs once at the beginning of the pipeline to map the SUT architecture.
+    """
+    print(f"\n{'='*60}")
+    print(f"🔍 DISCOVERY AGENT (Graphify) — Mapping Architecture")
+    print(f"{'='*60}")
+    
+    description = state.get("sut_description", "")
+    domain = state.get("domain", "")
+    
+    # If the architecture is already provided (e.g. by the adapter), skip LLM discovery
+    if state.get("sut_architecture"):
+        print("  Architecture already provided. Skipping LLM discovery.")
+        return {}
+        
+    print("  Analyzing SUT description to deduce internal graph...")
+    
+    llm = _get_llm()
+    prompt = DISCOVERY_USER_PROMPT.format(
+        name="Target System",
+        domain=domain,
+        description=description
+    )
+    
+    messages = [
+        SystemMessage(content=DISCOVERY_SYSTEM_PROMPT),
+        HumanMessage(content=prompt)
+    ]
+    
+    response = llm.invoke(messages)
+    architecture = response.content
+    
+    print("\n  🗺️ Deduced Architecture Graph:")
+    # Print the first few lines as a preview
+    preview = "\n".join([f"    {line}" for line in architecture.split("\n")[:10]])
+    print(f"{preview}\n    ...")
+    
+    return {
+        "sut_architecture": architecture
+    }

agentic_qa-0.1.0/agentic_qa/agents/executor.py

@@ -0,0 +1,89 @@
+"""
+Executor Agent — Generic System Under Test Runner.
+
+Executes test cases against ANY connected SUT (RAG, API, function).
+Uses the active SUT adapter from the registry — works with:
+  - Built-in Financial RAG demo
+  - Any RAG connected via API endpoint
+  - Any Python function wrapped as a callable
+"""
+
+import time
+from agentic_qa.graph.state import QAState
+from agentic_qa.sut import get_active_sut
+
+
+def executor_node(state: QAState) -> dict:
+    """
+    LangGraph node: Executor Agent.
+    
+    Runs each test case from the current iteration through whatever
+    SUT is currently active and collects results.
+    """
+    iteration = state.get("current_iteration", 1)
+    test_suite = state.get("test_suite", [])
+    
+    print(f"\n{'='*60}")
+    print(f"⚡ EXECUTOR AGENT — Iteration {iteration}")
+    print(f"{'='*60}")
+    
+    # Get test cases for the current iteration only
+    iter_prefix = f"TC-{iteration:02d}"
+    current_tests = [tc for tc in test_suite if tc["id"].startswith(iter_prefix)]
+    
+    if not current_tests:
+        num_per_iter = 5
+        current_tests = test_suite[-num_per_iter:]
+    
+    # Get the active SUT (whatever the user connected)
+    sut = get_active_sut()
+    print(f"  SUT: {sut.name}")
+    print(f"  Executing {len(current_tests)} test cases...")
+    
+    execution_results = []
+    
+    for tc in current_tests:
+        test_id = tc["id"]
+        input_data = tc["input_data"]
+        
+        print(f"  ▶ Running [{test_id}]...", end=" ")
+        
+        start_time = time.time()
+        try:
+            output = sut.process(input_data)
+            exec_time = time.time() - start_time
+            
+            # Normalize output — adapters return "output" key
+            sut_output = output.get("output", output.get("sut_output", str(output)))
+            status = output.get("status", "unknown")
+            
+            result = {
+                "test_id": test_id,
+                "sut_output": str(output),
+                "execution_time": round(exec_time, 4),
+                "error": output.get("error") if status == "error" else None,
+            }
+            print(f"Done ({exec_time:.3f}s) — Status: {status}")
+            
+        except Exception as e:
+            exec_time = time.time() - start_time
+            result = {
+                "test_id": test_id,
+                "sut_output": "",
+                "execution_time": round(exec_time, 4),
+                "error": str(e),
+            }
+            print(f"ERROR ({exec_time:.3f}s) — {e}")
+        
+        execution_results.append(result)
+    
+    errors = sum(1 for r in execution_results if r["error"])
+    avg_time = sum(r["execution_time"] for r in execution_results) / max(len(execution_results), 1)
+    print(f"\n  📊 Execution Summary:")
+    print(f"     Tests executed: {len(execution_results)}")
+    print(f"     Errors/crashes: {errors}")
+    print(f"     Avg exec time:  {avg_time:.3f}s")
+    
+    return {
+        "execution_results": execution_results,
+    }