bharatrag 0.1.0__tar.gz

bharatrag-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 pradnyagundu
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

bharatrag-0.1.0/PKG-INFO

@@ -0,0 +1,251 @@
+Metadata-Version: 2.4
+Name: bharatrag
+Version: 0.1.0
+Summary: RAG evaluation library for Indian languages (Hindi, Marathi)
+Author-email: Pradnya Gundu <gundupradnya@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/pradnyagundu/bharatrag
+Project-URL: Repository, https://github.com/pradnyagundu/bharatrag
+Keywords: RAG,evaluation,Hindi,Marathi,Indic,NLP,LLM
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: sentence-transformers>=2.2.0
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: scikit-learn>=1.2.0
+Requires-Dist: python-dotenv>=1.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Dynamic: license-file
+
+# bharatrag
+Open-source RAG evaluation library for Indian languages (Hindi, Marathi). pip install bharatrag
+# BharatRAG 🇮🇳
+
+**RAG Evaluation Library for Indian Languages**
+
+[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![GitHub](https://img.shields.io/badge/GitHub-bharatrag-black.svg)](https://github.com/pradnyagundu/bharatrag)
+
+BharatRAG is the first open-source RAG evaluation library built specifically for **Indian languages (Hindi and Marathi)**. 
+
+Existing tools like RAGAS are built and tested on English data. BharatRAG fills the gap — giving developers a reliable way to measure RAG quality in Indic languages.
+
+---
+
+## The Problem
+
+RAG (Retrieval Augmented Generation) systems are being deployed across India for:
+- Government scheme chatbots (PM Kisan, Ayushman Bharat)
+- Health information systems in regional languages
+- EdTech platforms for vernacular learners
+- Banking and insurance customer support
+
+But there is **no standard way to evaluate** whether these systems are actually working correctly in Hindi, Marathi, or other Indian languages.
+
+RAGAS — the most popular RAG evaluation tool — uses English-first embedding models that produce unreliable scores for Indic text.
+
+**BharatRAG solves this.**
+
+---
+
+## What it measures
+
+BharatRAG computes the **RAG Triad** in Hindi and Marathi:
+
+| Metric | Question it answers |
+|---|---|
+| **Context Relevance** | Did we retrieve the right documents? |
+| **Groundedness** | Is the answer based on the context, or hallucinated? |
+| **Answer Relevance** | Does the answer actually address the question? |
+
+---
+
+## Benchmark Results
+
+BharatRAG evaluated on 20 Hindi + Marathi QA pairs across government schemes, health, and agriculture domains:
+
+| Metric | Hindi ✅ Correct | Hindi ❌ Hallucinated | Marathi ✅ Correct | Marathi ❌ Hallucinated |
+|---|---|---|---|---|
+| Context Relevance | 0.4793 | 0.4793 | 0.4327 | 0.4327 |
+| Groundedness | 0.9167 | 0.7000 | 0.9667 | 0.2000 |
+| Answer Relevance | 0.6221 | 0.5417 | 0.5072 | 0.2959 |
+| **Overall** | **0.6727** | **0.5737** | **0.6355** | **0.3095** |
+
+BharatRAG correctly scores hallucinated answers **lower** than correct answers in both languages.
+Marathi hallucination detection shows a **2x difference** in overall score (0.6355 vs 0.3095).
+
+---
+
+## Installation
+
+```bash
+pip install bharatrag
+```
+
+---
+
+## Quick Start
+
+```python
+from bharatrag import evaluate
+
+results = evaluate(
+    questions=["पीएम किसान योजना में कितने रुपये मिलते हैं?"],
+    contexts=[[
+        "पीएम किसान सम्मान निधि योजना के तहत किसानों को",
+        "प्रति वर्ष 6000 रुपये तीन किश्तों में मिलते हैं।"
+    ]],
+    answers=["पीएम किसान योजना में किसानों को 6000 रुपये मिलते हैं।"],
+    language="hindi"
+)
+
+print(results)
+# {
+#   'context_relevance': 0.72,
+#   'groundedness': 1.0,
+#   'answer_relevance': 0.66,
+#   'overall': 0.79,
+#   'language': 'hindi',
+#   'num_questions': 1
+# }
+```
+
+### Marathi support
+
+```python
+results = evaluate(
+    questions=["पीएम किसान योजनेत किती रुपये मिळतात?"],
+    contexts=[[
+        "पीएम किसान सन्मान निधी योजनेंतर्गत शेतकऱ्यांना",
+        "दरवर्षी 6000 रुपये तीन हप्त्यांमध्ये मिळतात."
+    ]],
+    answers=["पीएम किसान योजनेत 6000 रुपये मिळतात."],
+    language="marathi"
+)
+```
+
+### Individual metrics
+
+```python
+from bharatrag.metrics.context_relevance import ContextRelevance
+from bharatrag.metrics.groundedness import Groundedness
+from bharatrag.metrics.answer_relevance import AnswerRelevance
+
+cr = ContextRelevance(language="hindi")
+score = cr.score(
+    question="भारत की राजधानी क्या है?",
+    contexts=["भारत की राजधानी नई दिल्ली है।"]
+)
+print(score)  # 0.61
+```
+
+---
+
+## Supported Languages
+
+| Language | Model Used |
+|---|---|
+| Hindi | `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2` |
+| Marathi | `l3cube-pune/marathi-sentence-bert-nli` |
+| English | `sentence-transformers/all-MiniLM-L6-v2` |
+
+More languages coming soon — Tamil, Bengali, Gujarati.
+
+---
+
+## Benchmark Dataset
+
+BharatRAG ships with a hand-curated benchmark dataset of **20 Hindi + Marathi QA pairs** across:
+- Government schemes (PM Kisan, Ayushman Bharat, Jan Dhan, Ujjwala)
+- Health (diabetes, sanitation)
+- Agriculture (wheat sowing, crop insurance)
+- Education (Mid Day Meal, Beti Bachao)
+
+Each example includes a correct answer and a hallucinated answer for evaluation testing.
+
+Dataset location: `data/benchmark.json`
+
+---
+
+## Project Structure
+
+bharatrag/
+├── bharatrag/
+│   ├── init.py          # evaluate() function
+│   ├── embeddings/
+│   │   └── indic_embeddings.py   # Indic embedding models
+│   └── metrics/
+│       ├── context_relevance.py  # Metric 1
+│       ├── groundedness.py       # Metric 2
+│       └── answer_relevance.py   # Metric 3
+├── data/
+│   └── benchmark.json       # 20 Hindi+Marathi QA pairs
+├── tests/
+│   └── test_metrics.py      # 21 pytest tests
+└── examples/
+└── run_benchmark.py     # Benchmark runner
+
+---
+
+## Running Tests
+
+```bash
+pip install pytest
+pytest tests/ -v
+# 21 passed ✅
+```
+
+---
+
+## Why BharatRAG?
+
+| Feature | RAGAS | BharatRAG |
+|---|---|---|
+| English RAG evaluation | ✅ | ✅ |
+| Hindi RAG evaluation | ❌ Unreliable | ✅ |
+| Marathi RAG evaluation | ❌ Not supported | ✅ |
+| Indic benchmark dataset | ❌ | ✅ |
+| Free, no API key needed | ✅ | ✅ |
+
+---
+
+## Roadmap
+
+- [x] Hindi support
+- [x] Marathi support
+- [x] 20-example benchmark dataset
+- [ ] Tamil support
+- [ ] Bengali support
+- [ ] 100-example benchmark dataset
+- [ ] LangChain integration
+- [ ] LlamaIndex integration
+- [ ] HuggingFace Spaces demo
+
+---
+
+## Author
+
+**Pradnya Gundu**
+B.E. Artificial Intelligence & Data Science, APCOER Pune
+
+- GitHub: [@pradnyagundu](https://github.com/pradnyagundu)
+- LinkedIn: [pradnya-gundu](https://linkedin.com/in/pradnya-gundu-b28737249)
+
+---
+
+## License
+
+MIT License — free to use, modify, and distribute.
+
+---
+
+## Contributing
+
+Contributions welcome! Especially:
+- New Indian language support
+- More benchmark QA pairs
+- Integration with LangChain/LlamaIndex
+
+Open an issue or submit a PR on GitHub.

bharatrag-0.1.0/README.md

@@ -0,0 +1,231 @@
+# bharatrag
+Open-source RAG evaluation library for Indian languages (Hindi, Marathi). pip install bharatrag
+# BharatRAG 🇮🇳
+
+**RAG Evaluation Library for Indian Languages**
+
+[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![GitHub](https://img.shields.io/badge/GitHub-bharatrag-black.svg)](https://github.com/pradnyagundu/bharatrag)
+
+BharatRAG is the first open-source RAG evaluation library built specifically for **Indian languages (Hindi and Marathi)**. 
+
+Existing tools like RAGAS are built and tested on English data. BharatRAG fills the gap — giving developers a reliable way to measure RAG quality in Indic languages.
+
+---
+
+## The Problem
+
+RAG (Retrieval Augmented Generation) systems are being deployed across India for:
+- Government scheme chatbots (PM Kisan, Ayushman Bharat)
+- Health information systems in regional languages
+- EdTech platforms for vernacular learners
+- Banking and insurance customer support
+
+But there is **no standard way to evaluate** whether these systems are actually working correctly in Hindi, Marathi, or other Indian languages.
+
+RAGAS — the most popular RAG evaluation tool — uses English-first embedding models that produce unreliable scores for Indic text.
+
+**BharatRAG solves this.**
+
+---
+
+## What it measures
+
+BharatRAG computes the **RAG Triad** in Hindi and Marathi:
+
+| Metric | Question it answers |
+|---|---|
+| **Context Relevance** | Did we retrieve the right documents? |
+| **Groundedness** | Is the answer based on the context, or hallucinated? |
+| **Answer Relevance** | Does the answer actually address the question? |
+
+---
+
+## Benchmark Results
+
+BharatRAG evaluated on 20 Hindi + Marathi QA pairs across government schemes, health, and agriculture domains:
+
+| Metric | Hindi ✅ Correct | Hindi ❌ Hallucinated | Marathi ✅ Correct | Marathi ❌ Hallucinated |
+|---|---|---|---|---|
+| Context Relevance | 0.4793 | 0.4793 | 0.4327 | 0.4327 |
+| Groundedness | 0.9167 | 0.7000 | 0.9667 | 0.2000 |
+| Answer Relevance | 0.6221 | 0.5417 | 0.5072 | 0.2959 |
+| **Overall** | **0.6727** | **0.5737** | **0.6355** | **0.3095** |
+
+BharatRAG correctly scores hallucinated answers **lower** than correct answers in both languages.
+Marathi hallucination detection shows a **2x difference** in overall score (0.6355 vs 0.3095).
+
+---
+
+## Installation
+
+```bash
+pip install bharatrag
+```
+
+---
+
+## Quick Start
+
+```python
+from bharatrag import evaluate
+
+results = evaluate(
+    questions=["पीएम किसान योजना में कितने रुपये मिलते हैं?"],
+    contexts=[[
+        "पीएम किसान सम्मान निधि योजना के तहत किसानों को",
+        "प्रति वर्ष 6000 रुपये तीन किश्तों में मिलते हैं।"
+    ]],
+    answers=["पीएम किसान योजना में किसानों को 6000 रुपये मिलते हैं।"],
+    language="hindi"
+)
+
+print(results)
+# {
+#   'context_relevance': 0.72,
+#   'groundedness': 1.0,
+#   'answer_relevance': 0.66,
+#   'overall': 0.79,
+#   'language': 'hindi',
+#   'num_questions': 1
+# }
+```
+
+### Marathi support
+
+```python
+results = evaluate(
+    questions=["पीएम किसान योजनेत किती रुपये मिळतात?"],
+    contexts=[[
+        "पीएम किसान सन्मान निधी योजनेंतर्गत शेतकऱ्यांना",
+        "दरवर्षी 6000 रुपये तीन हप्त्यांमध्ये मिळतात."
+    ]],
+    answers=["पीएम किसान योजनेत 6000 रुपये मिळतात."],
+    language="marathi"
+)
+```
+
+### Individual metrics
+
+```python
+from bharatrag.metrics.context_relevance import ContextRelevance
+from bharatrag.metrics.groundedness import Groundedness
+from bharatrag.metrics.answer_relevance import AnswerRelevance
+
+cr = ContextRelevance(language="hindi")
+score = cr.score(
+    question="भारत की राजधानी क्या है?",
+    contexts=["भारत की राजधानी नई दिल्ली है।"]
+)
+print(score)  # 0.61
+```
+
+---
+
+## Supported Languages
+
+| Language | Model Used |
+|---|---|
+| Hindi | `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2` |
+| Marathi | `l3cube-pune/marathi-sentence-bert-nli` |
+| English | `sentence-transformers/all-MiniLM-L6-v2` |
+
+More languages coming soon — Tamil, Bengali, Gujarati.
+
+---
+
+## Benchmark Dataset
+
+BharatRAG ships with a hand-curated benchmark dataset of **20 Hindi + Marathi QA pairs** across:
+- Government schemes (PM Kisan, Ayushman Bharat, Jan Dhan, Ujjwala)
+- Health (diabetes, sanitation)
+- Agriculture (wheat sowing, crop insurance)
+- Education (Mid Day Meal, Beti Bachao)
+
+Each example includes a correct answer and a hallucinated answer for evaluation testing.
+
+Dataset location: `data/benchmark.json`
+
+---
+
+## Project Structure
+
+bharatrag/
+├── bharatrag/
+│   ├── init.py          # evaluate() function
+│   ├── embeddings/
+│   │   └── indic_embeddings.py   # Indic embedding models
+│   └── metrics/
+│       ├── context_relevance.py  # Metric 1
+│       ├── groundedness.py       # Metric 2
+│       └── answer_relevance.py   # Metric 3
+├── data/
+│   └── benchmark.json       # 20 Hindi+Marathi QA pairs
+├── tests/
+│   └── test_metrics.py      # 21 pytest tests
+└── examples/
+└── run_benchmark.py     # Benchmark runner
+
+---
+
+## Running Tests
+
+```bash
+pip install pytest
+pytest tests/ -v
+# 21 passed ✅
+```
+
+---
+
+## Why BharatRAG?
+
+| Feature | RAGAS | BharatRAG |
+|---|---|---|
+| English RAG evaluation | ✅ | ✅ |
+| Hindi RAG evaluation | ❌ Unreliable | ✅ |
+| Marathi RAG evaluation | ❌ Not supported | ✅ |
+| Indic benchmark dataset | ❌ | ✅ |
+| Free, no API key needed | ✅ | ✅ |
+
+---
+
+## Roadmap
+
+- [x] Hindi support
+- [x] Marathi support
+- [x] 20-example benchmark dataset
+- [ ] Tamil support
+- [ ] Bengali support
+- [ ] 100-example benchmark dataset
+- [ ] LangChain integration
+- [ ] LlamaIndex integration
+- [ ] HuggingFace Spaces demo
+
+---
+
+## Author
+
+**Pradnya Gundu**
+B.E. Artificial Intelligence & Data Science, APCOER Pune
+
+- GitHub: [@pradnyagundu](https://github.com/pradnyagundu)
+- LinkedIn: [pradnya-gundu](https://linkedin.com/in/pradnya-gundu-b28737249)
+
+---
+
+## License
+
+MIT License — free to use, modify, and distribute.
+
+---
+
+## Contributing
+
+Contributions welcome! Especially:
+- New Indian language support
+- More benchmark QA pairs
+- Integration with LangChain/LlamaIndex
+
+Open an issue or submit a PR on GitHub.

bharatrag-0.1.0/bharatrag/__init__.py

@@ -0,0 +1,77 @@
+"""
+BharatRAG — RAG Evaluation Library for Indian Languages
+Author: Pradnya Gundu
+"""
+
+from bharatrag.embeddings.indic_embeddings import IndicEmbedder
+from bharatrag.metrics.context_relevance import ContextRelevance
+from bharatrag.metrics.groundedness import Groundedness
+from bharatrag.metrics.answer_relevance import AnswerRelevance
+
+__version__ = "0.1.0"
+__author__ = "Pradnya Gundu"
+
+
+def evaluate(questions, contexts, answers, language="hindi"):
+    """
+    Evaluate a RAG system on Indian language data.
+
+    Args:
+        questions: list of questions (in Hindi/Marathi)
+        contexts:  list of lists — retrieved context chunks per question
+        answers:   list of generated answers
+        language:  "hindi" or "marathi" (default: "hindi")
+
+    Returns:
+        dict with scores for each metric
+
+    Example:
+        >>> from bharatrag import evaluate
+        >>> results = evaluate(
+        ...     questions=["भारत की राजधानी क्या है?"],
+        ...     contexts=[["भारत की राजधानी नई दिल्ली है।"]],
+        ...     answers=["भारत की राजधानी नई दिल्ली है।"],
+        ...     language="hindi"
+        ... )
+        >>> print(results)
+    """
+    print(f"\nLoading embedding model for {language}...")
+
+    # Load embedder ONCE and share across all 3 metrics
+    embedder = IndicEmbedder(language=language)
+
+    # Pass the same embedder to all metrics — no reloading
+    cr = ContextRelevance(language=language, embedder=embedder)
+    gr = Groundedness(language=language, embedder=embedder)
+    ar = AnswerRelevance(language=language, embedder=embedder)
+
+    print(f"Evaluating {len(questions)} question(s) in {language}...")
+
+    cr_scores = []
+    gr_scores = []
+    ar_scores = []
+
+    for i, (question, context, answer) in enumerate(
+        zip(questions, contexts, answers)
+    ):
+        print(f"  Scoring question {i+1}/{len(questions)}...")
+        cr_scores.append(cr.score(question, context))
+        gr_scores.append(gr.score(answer, context))
+        ar_scores.append(ar.score(question, answer))
+
+    results = {
+        "context_relevance": round(sum(cr_scores) / len(cr_scores), 4),
+        "groundedness":      round(sum(gr_scores) / len(gr_scores), 4),
+        "answer_relevance":  round(sum(ar_scores) / len(ar_scores), 4),
+        "language":          language,
+        "num_questions":     len(questions),
+    }
+
+    results["overall"] = round(
+        (results["context_relevance"] +
+         results["groundedness"] +
+         results["answer_relevance"]) / 3,
+        4
+    )
+
+    return results

bharatrag-0.1.0/bharatrag/embeddings/indic_embeddings.py

@@ -0,0 +1,110 @@
+"""
+Indic Embeddings — loads multilingual embedding models
+that actually understand Hindi and Marathi.
+"""
+
+from sentence_transformers import SentenceTransformer
+import numpy as np
+from sklearn.metrics.pairwise import cosine_similarity
+
+
+# Best free models for Indian languages
+INDIC_MODELS = {
+    "hindi": "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
+    "marathi": "l3cube-pune/marathi-sentence-bert-nli",
+    "english": "sentence-transformers/all-MiniLM-L6-v2",
+}
+
+
+class IndicEmbedder:
+    """
+    Loads the right embedding model for a given Indian language
+    and computes sentence embeddings + similarity scores.
+
+    Example:
+        >>> embedder = IndicEmbedder(language="hindi")
+        >>> score = embedder.similarity(
+        ...     "भारत की राजधानी क्या है?",
+        ...     "भारत की राजधानी नई दिल्ली है।"
+        ... )
+        >>> print(score)  # 0.91
+    """
+
+    def __init__(self, language: str = "hindi"):
+        """
+        Args:
+            language: "hindi", "marathi", or "english"
+        """
+        if language not in INDIC_MODELS:
+            raise ValueError(
+                f"Language '{language}' not supported. "
+                f"Choose from: {list(INDIC_MODELS.keys())}"
+            )
+
+        self.language = language
+        self.model_name = INDIC_MODELS[language]
+
+        print(f"Loading embedding model for {language}: {self.model_name}")
+        self.model = SentenceTransformer(self.model_name)
+        print("Model loaded successfully!")
+
+    def embed(self, text: str) -> np.ndarray:
+        """
+        Convert a single text string into a vector (embedding).
+
+        Args:
+            text: input string in Hindi/Marathi/English
+
+        Returns:
+            numpy array of shape (embedding_dim,)
+        """
+        return self.model.encode(text, convert_to_numpy=True)
+
+    def embed_batch(self, texts: list) -> np.ndarray:
+        """
+        Convert a list of texts into embeddings all at once.
+        Faster than calling embed() in a loop.
+
+        Args:
+            texts: list of strings
+
+        Returns:
+            numpy array of shape (len(texts), embedding_dim)
+        """
+        return self.model.encode(texts, convert_to_numpy=True)
+
+    def similarity(self, text1: str, text2: str) -> float:
+        """
+        Compute similarity between two texts.
+        Returns a score between 0 and 1.
+        1.0 = identical meaning, 0.0 = completely unrelated.
+
+        Args:
+            text1: first string
+            text2: second string
+
+        Returns:
+            float between 0 and 1
+        """
+        emb1 = self.embed(text1).reshape(1, -1)
+        emb2 = self.embed(text2).reshape(1, -1)
+        score = cosine_similarity(emb1, emb2)[0][0]
+        # Clip to [0, 1] — cosine can return tiny negatives
+        return float(np.clip(score, 0.0, 1.0))
+
+    def similarity_one_to_many(self, query: str, candidates: list) -> list:
+        """
+        Compare one query against many candidate texts.
+        Used in Context Relevance: compare question vs all chunks.
+
+        Args:
+            query: the question
+            candidates: list of context chunks
+
+        Returns:
+            list of similarity scores, one per candidate
+        """
+        query_emb = self.embed(query).reshape(1, -1)
+        candidate_embs = self.embed_batch(candidates)
+        scores = cosine_similarity(query_emb, candidate_embs)[0]
+        return [float(np.clip(s, 0.0, 1.0)) for s in scores]