llmgrader 1.0.0__tar.gz

llmgrader-1.0.0/LICENSE

@@ -0,0 +1,59 @@
+Apache License
+Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+"License" shall mean the terms and conditions for use, reproduction, and distribution.
+
+"Licensor" shall mean the copyright owner or entity authorized by the copyright owner.
+
+"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity.
+
+"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.
+
+"Source" form shall mean the preferred form for making modifications.
+
+"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form.
+
+"Work" shall mean the work of authorship made available under the License.
+
+"Derivative Works" shall mean any work that is based on the Work.
+
+"Contribution" shall mean any work of authorship submitted to the Licensor for inclusion in the Work.
+
+"Contributor" shall mean Licensor and any Legal Entity on behalf of whom a Contribution has been received by the Licensor and included within the Work.
+
+2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work.
+
+4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
+   (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and
+   (b) You must cause any modified files to carry prominent notices stating that You changed the files; and
+   (c) You must retain, in all Source forms of the Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work; and
+   (d) If the Work includes a "NOTICE" text file, You must include a readable copy of the attribution notices contained within such NOTICE file.
+
+5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License.
+
+6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor.
+
+7. Disclaimer of Warranty. UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING, LICENSOR PROVIDES THE WORK ON AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND.
+
+8. Limitation of Liability. IN NO EVENT SHALL ANY CONTRIBUTOR BE LIABLE FOR ANY DAMAGES ARISING FROM THIS LICENSE OR OUT OF OR IN CONNECTION WITH THE USE OR INABILITY TO USE THE WORK.
+
+Copyright 2024 Mahesh Makvana
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

llmgrader-1.0.0/PKG-INFO

@@ -0,0 +1,452 @@
+Metadata-Version: 2.4
+Name: llmgrader
+Version: 1.0.0
+Summary: Open-source LLM evaluation framework — 50+ research-backed metrics for RAG, agents, safety, and more
+Home-page: https://github.com/maheshmakvana/llmeval
+Author: Mahesh Makvana
+Author-email: maheshmakvana@example.com
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: openai>=1.0.0
+Requires-Dist: anthropic>=0.20.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: typer>=0.9.0
+Requires-Dist: pytest>=7.0.0
+Requires-Dist: httpx>=0.24.0
+Requires-Dist: numpy>=1.20.0
+Requires-Dist: tenacity>=8.0.0
+Requires-Dist: jinja2>=3.0.0
+Requires-Dist: colorama>=0.4.6
+Provides-Extra: langchain
+Requires-Dist: langchain>=0.1.0; extra == "langchain"
+Requires-Dist: langchain-openai>=0.1.0; extra == "langchain"
+Provides-Extra: llamaindex
+Requires-Dist: llama-index>=0.10.0; extra == "llamaindex"
+Provides-Extra: ollama
+Requires-Dist: ollama>=0.1.0; extra == "ollama"
+Provides-Extra: all
+Requires-Dist: langchain>=0.1.0; extra == "all"
+Requires-Dist: langchain-openai>=0.1.0; extra == "all"
+Requires-Dist: llama-index>=0.10.0; extra == "all"
+Requires-Dist: ollama>=0.1.0; extra == "all"
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# llmeval
+
+**Open-source LLM evaluation framework** — 50+ research-backed metrics for RAG pipelines, AI agents, safety, and conversational systems. Pytest-native. Provider-agnostic.
+
+```bash
+pip install llmeval
+```
+
+---
+
+## Quick Start
+
+```python
+from llmeval import LLMTestCase, assert_test
+from llmeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
+
+tc = LLMTestCase(
+    input="What is the capital of France?",
+    actual_output="The capital of France is Paris.",
+    retrieval_context=["France is a country in Western Europe. Its capital is Paris."],
+)
+
+assert_test(tc, metrics=[
+    AnswerRelevancyMetric(threshold=0.7),
+    FaithfulnessMetric(threshold=0.8),
+])
+```
+
+---
+
+## Features
+
+| Category | What it does |
+|---|---|
+| **RAG Evaluation** | Answer relevancy, faithfulness, contextual precision/recall/relevancy |
+| **Custom Metrics** | GEval (LLM-as-judge + CoT), DAG (deterministic decision-tree) |
+| **Safety** | Hallucination, bias, toxicity, PII leakage, misuse detection |
+| **Agent Evaluation** | Task completion, tool correctness, step efficiency, argument correctness |
+| **Conversational** | Relevancy, completeness, role adherence, knowledge retention |
+| **Other** | JSON correctness (with schema), summarization quality |
+| **Pytest Native** | `assert_test()`, fixtures, parametrize helpers |
+| **Tracing** | `@observe` decorator, span/trace trees, component-level evaluation |
+| **Bulk Evaluation** | `evaluate()` with concurrent execution and aggregated reports |
+| **Dataset Tools** | `EvaluationDataset`, versioned JSON storage, CSV import |
+| **Synthesizer** | Auto-generate Goldens from documents (4-step pipeline) |
+| **Providers** | OpenAI, Azure OpenAI, Anthropic, Ollama, custom LLM base class |
+| **Integrations** | LangChain, LlamaIndex, CrewAI |
+| **CLI** | `llmeval test`, `llmeval set-openai`, `llmeval list-metrics` |
+
+---
+
+## Installation
+
+```bash
+# Core
+pip install llmeval
+
+# With LangChain integration
+pip install "llmeval[langchain]"
+
+# With LlamaIndex
+pip install "llmeval[llamaindex]"
+
+# Everything
+pip install "llmeval[all]"
+```
+
+---
+
+## Metrics Reference
+
+### RAG Metrics
+
+```python
+from llmeval.metrics import (
+    AnswerRelevancyMetric,       # Is the answer relevant to the question?
+    FaithfulnessMetric,          # Are claims grounded in retrieved context?
+    ContextualRelevancyMetric,   # Are retrieved chunks relevant to the query?
+    ContextualPrecisionMetric,   # Do relevant chunks rank higher?
+    ContextualRecallMetric,      # Does context cover expected answer claims?
+)
+
+tc = LLMTestCase(
+    input="What causes rain?",
+    actual_output="Rain is caused by water vapor condensing in clouds.",
+    expected_output="Rain is caused by condensation of water vapor.",
+    retrieval_context=[
+        "The water cycle involves evaporation and condensation.",
+        "Rain forms when water vapor cools and condenses around particles.",
+    ],
+)
+
+result = FaithfulnessMetric(threshold=0.8).measure(tc)
+print(result.score, result.reason)
+```
+
+### Custom: GEval (LLM-as-Judge)
+
+```python
+from llmeval import GEvalMetric, LLMTestCaseParams
+
+metric = GEvalMetric(
+    name="Correctness",
+    criteria="The output should be factually correct and directly answer the question.",
+    evaluation_params=[
+        LLMTestCaseParams.INPUT,
+        LLMTestCaseParams.ACTUAL_OUTPUT,
+        LLMTestCaseParams.EXPECTED_OUTPUT,
+    ],
+    threshold=0.7,
+)
+
+result = metric.measure(tc)
+```
+
+### Custom: DAG (Deterministic)
+
+```python
+from llmeval import DAGMetric
+from llmeval.metrics.custom.dag import DAGNode
+
+dag = DAGNode(
+    condition=lambda tc: len(tc.actual_output) > 0,
+    score_if_false=0.0,
+    next_if_true=DAGNode(
+        condition=lambda tc: "error" not in tc.actual_output.lower(),
+        score_if_true=1.0,
+        score_if_false=0.2,
+    )
+)
+metric = DAGMetric(name="ResponseQuality", root=dag, threshold=0.5)
+```
+
+### Safety Metrics
+
+```python
+from llmeval.metrics import (
+    HallucinationMetric,   # Detects factual hallucinations vs context
+    BiasMetric,            # Gender, racial, political, religious bias
+    ToxicityMetric,        # Hate speech, harassment, harmful content
+    PIILeakageMetric,      # SSN, email, phone, credit card detection
+    MisuseMetric,          # Weapons, illegal activity enablement
+)
+
+result = BiasMetric(threshold=0.7).measure(tc)
+```
+
+### Agentic Metrics
+
+```python
+from llmeval import ToolCall
+from llmeval.metrics import (
+    TaskCompletionMetric,      # Did the agent accomplish the goal?
+    ToolCorrectnessMetric,     # Were the right tools called?
+    StepEfficiencyMetric,      # Were unnecessary steps avoided?
+    ArgumentCorrectnessMetric, # Were tool arguments correct?
+)
+
+tc = LLMTestCase(
+    input="Search for the latest news on AI and summarize it.",
+    actual_output="Here is a summary of recent AI news...",
+    tools_called=[
+        ToolCall(name="web_search", input_parameters={"query": "latest AI news"}),
+        ToolCall(name="summarize", input_parameters={"max_length": 200}),
+    ],
+    expected_tools=["web_search", "summarize"],
+)
+
+result = ToolCorrectnessMetric(threshold=0.8).measure(tc)
+```
+
+### Conversational Metrics
+
+```python
+from llmeval import ConversationalTestCase, Message
+from llmeval.metrics import (
+    ConversationalRelevancyMetric,
+    ConversationCompletenessMetric,
+    RoleAdherenceMetric,
+    KnowledgeRetentionMetric,
+)
+
+tc = ConversationalTestCase(
+    messages=[
+        Message(role="user", content="My name is Alice and I like Python."),
+        Message(role="assistant", content="Nice to meet you, Alice! Python is great."),
+        Message(role="user", content="What's my name again?"),
+        Message(role="assistant", content="Your name is Alice."),
+    ],
+    chatbot_role="A helpful assistant that remembers user preferences.",
+)
+
+result = KnowledgeRetentionMetric(threshold=0.7).measure(tc)
+```
+
+---
+
+## Bulk Evaluation
+
+```python
+from llmeval import evaluate, LLMTestCase
+from llmeval.metrics import AnswerRelevancyMetric, JSONCorrectnessMetric
+
+test_cases = [
+    LLMTestCase(input="What is 2+2?", actual_output="4"),
+    LLMTestCase(input="Capital of Japan?", actual_output="Tokyo"),
+    LLMTestCase(input="Return JSON", actual_output='{"status": "ok"}'),
+]
+
+result = evaluate(
+    test_cases=test_cases,
+    metrics=[AnswerRelevancyMetric(), JSONCorrectnessMetric()],
+    max_concurrent=4,
+    verbose=True,
+)
+
+print(f"Pass rate: {result.pass_rate:.1%}")
+print(f"Overall score: {result.overall_score:.3f}")
+result.print_summary()
+```
+
+---
+
+## Pytest Integration
+
+```python
+# test_my_llm.py
+import pytest
+from llmeval import LLMTestCase, assert_test
+from llmeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
+
+def test_rag_answer():
+    tc = LLMTestCase(
+        input="What causes lightning?",
+        actual_output=my_rag_pipeline("What causes lightning?"),
+        retrieval_context=get_context("lightning"),
+    )
+    assert_test(tc, metrics=[
+        AnswerRelevancyMetric(threshold=0.7),
+        FaithfulnessMetric(threshold=0.8),
+    ])
+
+
+# Run with: llmeval test test_my_llm.py
+# Or:       pytest test_my_llm.py
+```
+
+---
+
+## Tracing & Component-Level Evaluation
+
+```python
+from llmeval import observe, Tracer, set_tracer, clear_tracer
+
+tracer = Tracer()
+set_tracer(tracer)
+trace = tracer.start_trace()
+
+@observe(span_type="retriever")
+def retrieve(query: str) -> list:
+    return vector_db.search(query)
+
+@observe(span_type="llm")
+def generate(context: list, query: str) -> str:
+    return llm.generate(f"Context: {context}\nQuestion: {query}")
+
+def rag_pipeline(query: str) -> str:
+    context = retrieve(query)
+    return generate(context, query)
+
+answer = rag_pipeline("What is quantum computing?")
+tracer.end_trace()
+clear_tracer()
+
+tracer.print_last_trace()  # Shows span tree with latencies
+```
+
+---
+
+## Dataset Management
+
+```python
+from llmeval import EvaluationDataset, Golden
+
+# Build a dataset
+ds = EvaluationDataset()
+ds.add_goldens([
+    Golden(input="What is AI?", expected_output="Artificial intelligence."),
+    Golden(input="Capital of Germany?", expected_output="Berlin"),
+])
+ds.save("my_dataset.json")
+
+# Load and use
+ds = EvaluationDataset.load("my_dataset.json")
+test_cases = ds.to_test_cases(generate_fn=my_llm.generate)
+```
+
+---
+
+## Synthetic Dataset Generation
+
+```python
+from llmeval import Synthesizer
+
+synth = Synthesizer()
+
+docs = [
+    "The Python programming language was created by Guido van Rossum...",
+    "Machine learning is a branch of artificial intelligence...",
+]
+
+goldens = synth.generate_goldens_from_docs(
+    documents=docs,
+    max_goldens_per_doc=5,
+    filter_questions=True,
+    evolve_questions=True,
+    generate_expected_outputs=True,
+)
+
+print(f"Generated {len(goldens)} golden test cases")
+for g in goldens[:2]:
+    print(f"Q: {g.input}")
+    print(f"A: {g.expected_output}\n")
+```
+
+---
+
+## LLM Providers
+
+```python
+from llmeval.providers import OpenAIProvider, AnthropicProvider, OllamaProvider
+
+# OpenAI
+provider = OpenAIProvider(model="gpt-4o", api_key="sk-...")
+
+# Anthropic Claude
+provider = AnthropicProvider(model="claude-sonnet-4-6")
+
+# Ollama (local)
+provider = OllamaProvider(model="llama3")
+
+# Custom provider
+from llmeval.providers import LLMProvider
+
+class MyProvider(LLMProvider):
+    def generate(self, prompt: str, **kwargs) -> str:
+        return my_llm_api.call(prompt)
+
+# Use in any metric
+metric = AnswerRelevancyMetric(model=MyProvider())
+```
+
+---
+
+## LangChain Integration
+
+```python
+from langchain_openai import ChatOpenAI
+from llmeval.integrations.langchain import LangChainCallbackHandler, evaluate_chain
+from llmeval.metrics import AnswerRelevancyMetric
+
+llm = ChatOpenAI(model="gpt-4o")
+chain = llm  # or any LangChain runnable
+
+result = evaluate_chain(
+    chain=chain,
+    inputs=["What is the capital of France?", "Who invented Python?"],
+    metrics=[AnswerRelevancyMetric(threshold=0.7)],
+)
+```
+
+---
+
+## CLI
+
+```bash
+# Run evaluation tests
+llmeval test tests/test_llm.py
+llmeval test tests/ -n 4          # 4 parallel workers
+
+# Configure providers
+llmeval set-openai --key sk-... --model gpt-4o
+llmeval set-anthropic --key sk-... --model claude-sonnet-4-6
+llmeval set-ollama --model llama3
+
+# List all metrics
+llmeval list-metrics
+
+# Version
+llmeval version
+```
+
+---
+
+## License
+
+Apache 2.0 — see LICENSE.
+
+**Author:** Mahesh Makvana