TruthTorchLM 0.1.0__tar.gz

truthtorchlm-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Yavuz Faruk Bakman
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

truthtorchlm-0.1.0/LICENSE copy

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Yavuz Faruk Bakman
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

truthtorchlm-0.1.0/PKG-INFO

@@ -0,0 +1,209 @@
+Metadata-Version: 2.1
+Name: TruthTorchLM
+Version: 0.1.0
+Summary: TruthTorchLM is an open-source library designed to detect and mitigate hallucinations in text generation models. The library integrates state-of-the-art methods, offers comprehensive benchmarking tools across various tasks, and enables seamless integration with popular frameworks like Huggingface and LiteLLM.
+Author: Yavuz Faruk Bakman
+Author-email: ybakman@usc.edu
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: LICENSE copy
+Requires-Dist: aiohttp==3.9.3
+Requires-Dist: evaluate==0.4.1
+Requires-Dist: instructor
+Requires-Dist: litellm==1.57.0
+Requires-Dist: nest_asyncio==1.6.0
+Requires-Dist: numpy==1.26.4
+Requires-Dist: outlines==0.1.4
+Requires-Dist: pandas==2.2.3
+Requires-Dist: pydantic
+Requires-Dist: PyYAML
+Requires-Dist: Requests==2.32.3
+Requires-Dist: scikit_learn==1.4.1.post1
+Requires-Dist: scipy==1.13.0
+Requires-Dist: sentence_transformers==3.1.1
+Requires-Dist: termcolor==2.5.0
+Requires-Dist: torch==2.2.2
+Requires-Dist: tqdm==4.66.2
+Requires-Dist: transformers==4.44.2
+Requires-Dist: absl-py
+Requires-Dist: nltk
+Requires-Dist: rouge_score
+
+# TruthTorchLM: A Comprehensive Library for Hallucination Detection in LLMs  
+
+**TruthTorchLM** is an open-source library designed to detect and mitigate hallucinations in text generation models. The library integrates state-of-the-art methods, offers comprehensive benchmarking tools across various tasks, and enables seamless integration with popular frameworks like Huggingface and LiteLLM.
+
+---
+
+## Features  
+
+- **State-of-the-Art Methods**: Implementations of advanced hallucination detection techniques.  
+- **Evaluation Tools**: Benchmark hallucination detection methods using various metrics like AUROC, PRR, and Accuracy.  
+- **Calibration**: Normalize and calibrate truth values for interpretable and comparable hallucination scores.  
+- **Integration**: Seamlessly works with Huggingface and LiteLLM.  
+- **Long-Form Generation**: Adapts detection methods to handle long-form text generations effectively.  
+- **Extendability**: Provides an intuitive interface for implementing new hallucination detection methods.  
+
+---
+
+## Installation  
+
+Install TruthTorchLM using pip:  
+
+```bash
+pip install TruthTorchLM
+```
+
+---
+
+## Quick Start  
+
+### Setting Up a Model  
+
+You can define your model and tokenizer using Huggingface or specify an API-based model:  
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import TruthTorchLM as ttlm
+import torch
+
+# Huggingface model
+model = AutoModelForCausalLM.from_pretrained(
+    "meta-llama/Llama-2-7b-chat-hf", 
+    torch_dtype=torch.bfloat16
+).to('cuda:0')
+tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf", use_fast=False)
+
+# API model
+api_model = "gpt-4o"
+```
+
+### Generating Text with Truth Values  
+
+TruthTorchLM generates messages with a truth value, indicating whether the model output is hallucinating or not. Various methods (called **truth methods**) can be used to detect hallucinations. Each method can have different algorithms and output ranges. Lower truth values generally suggest hallucinations. This functionality is mostly useful for short-form QA:
+```python
+# Define truth methods
+lars = ttlm.truth_methods.LARS()
+confidence = ttlm.truth_methods.Confidence()
+self_detection = ttlm.truth_methods.SelfDetection(number_of_questions=5)
+truth_methods = [lars, confidence, self_detection]
+```
+```python
+# Define a chat history
+chat = [{"role": "system", "content": "You are a helpful assistant. Give short and precise answers."},
+        {"role": "user", "content": "What is the capital city of France?"}]
+```
+```python
+# Generate text with truth values (Huggingface model)
+output_hf_model = ttlm.generate_with_truth_value(
+    model=model,
+    tokenizer=tokenizer,
+    messages=chat,
+    truth_methods=truth_methods,
+    max_new_tokens=100,
+    temperature=0.7
+)
+# Generate text with truth values (API model)
+output_api_model = ttlm.generate_with_truth_value(
+    model=api_model,
+    messages=chat,
+    truth_methods=truth_methods
+)
+```
+
+### Calibrating Truth Methods  
+Truth values for different methods may not be directly comparable. Use the `calibrate_truth_method` function to normalize truth values to a common range for better interpretability
+
+```python
+model_judge = ttlm.evaluators.ModelJudge('gpt-4o-mini')
+calibration_results = ttlm.calibrate_truth_method(
+    dataset='trivia_qa',
+    model=model,
+    truth_methods=truth_methods,
+    tokenizer=tokenizer,
+    correctness_evaluator=model_judge,
+    size_of_data=1000,
+    max_new_tokens=64
+)
+```
+
+### Evaluating Truth Methods  
+
+We can evaluate the truth methods with the `evaluate_truth_method` function. We can define different evaluation metrics including AUROC, AUPRC, AUARC, Accuracy, F1, Precision, Recall, PRR:  
+
+```python
+results = ttlm.evaluate_truth_method(
+    dataset='trivia_qa',
+    model=model,
+    truth_methods=truth_methods,
+    eval_metrics=['auroc', 'prr'],
+    tokenizer=tokenizer,
+    size_of_data=1000,
+    correctness_evaluator=model_judge,
+    max_new_tokens=64
+)
+```
+
+---
+
+## Available Hallucination Detection Methods  
+
+- **LARS**: [Do Not Design, Learn: A Trainable Scoring Function for Uncertainty Estimation in Generative LLMs](https://arxiv.org/pdf/2406.11278).  
+- **Confidence**: [Uncertainty Estimation in Autoregressive Structured Prediction](https://openreview.net/pdf?id=jN5y-zb5Q7m). 
+- **Entropy**:[Uncertainty Estimation in Autoregressive Structured Prediction](https://openreview.net/pdf?id=jN5y-zb5Q7m).  
+- **SelfDetection**: [Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Method](https://arxiv.org/pdf/2310.17918). 
+- **AttentionScore**: [LLM-Check: Investigating Detection of Hallucinations in Large Language Models](https://openreview.net/pdf?id=LYx4w3CAgy). 
+- **CrossExamination**: [LM vs LM: Detecting Factual Errors via Cross Examination](https://arxiv.org/pdf/2305.13281). 
+- **EccentricityConfidence**: [Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models](https://arxiv.org/pdf/2305.19187). 
+- **EccentricityUncertainty**: [Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models](https://arxiv.org/pdf/2305.19187).
+- **GoogleSearchCheck**: [FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios](https://arxiv.org/pdf/2307.13528). 
+- **Inside**: [INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection](https://openreview.net/pdf?id=Zj12nzlQbz). 
+- **KernelLanguageEntropy**: [Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities](https://arxiv.org/pdf/2405.20003). 
+- **MARS**: [MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMs](https://aclanthology.org/2024.acl-long.419.pdf). 
+- **MatrixDegreeConfidence**: [Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models](https://arxiv.org/pdf/2305.19187). 
+- **MatrixDegreeUncertainty**: [Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models](https://arxiv.org/pdf/2305.19187).
+- **MultiLLMCollab**: [Don’t Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration](https://arxiv.org/pdf/2402.00367). 
+- **NumSemanticSetUncertainty**: [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://arxiv.org/pdf/2302.09664). 
+- **PTrue**: [Language Models (Mostly) Know What They Know](https://arxiv.org/pdf/2207.05221). 
+- **Saplma**: [The Internal State of an LLM Knows When It’s Lying](https://aclanthology.org/2023.findings-emnlp.68.pdf). 
+- **SemanticEntropy**: [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://arxiv.org/pdf/2302.09664).
+- **sentSAR**: [Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models](https://aclanthology.org/2024.acl-long.276.pdf). 
+- **SumEigenUncertainty**: [Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models](https://arxiv.org/pdf/2305.19187).
+- **tokenSAR**: [Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models](https://aclanthology.org/2024.acl-long.276.pdf). 
+- **VerbalizedConfidence**: [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://openreview.net/pdf?id=g3faCfrwm7). 
+
+---
+
+## Contributors  
+
+- **Yavuz Faruk Bakman** (ybakman@usc.edu)  
+- **Duygu Nur Yaldiz** (yaldiz@usc.edu)  
+- **Sungmin Kang** (kangsung@usc.edu)  
+- **Hayrettin Eren Yildiz** (hayereyil@gmail.com)  
+- **Alperen Ozis** (alperenozis@gmail.com)  
+
+---
+
+## Citation  
+
+If you use TruthTorchLM in your research, please cite:  
+
+```bibtex
+@misc{truthtorchlm2025,
+  title={TruthTorchLM: A Comprehensive Library for Hallucination Detection in Large Language Models},
+  author={Yavuz Faruk Bakman, Duygu Nur Yaldiz,Sungmin Kang, Hayrettin Eren Yildiz, Alperen Ozis},
+  year={2025},
+  howpublished={GitHub},
+  url={https://github.com/Ybakman/TruthTorchLM}
+}
+```
+
+---
+
+## License  
+
+TruthTorchLM is released under the [MIT License](LICENSE).  
+
+For inquiries or support, feel free to contact the maintainers.
+

truthtorchlm-0.1.0/README.md

@@ -0,0 +1,177 @@
+# TruthTorchLM: A Comprehensive Library for Hallucination Detection in LLMs  
+
+**TruthTorchLM** is an open-source library designed to detect and mitigate hallucinations in text generation models. The library integrates state-of-the-art methods, offers comprehensive benchmarking tools across various tasks, and enables seamless integration with popular frameworks like Huggingface and LiteLLM.
+
+---
+
+## Features  
+
+- **State-of-the-Art Methods**: Implementations of advanced hallucination detection techniques.  
+- **Evaluation Tools**: Benchmark hallucination detection methods using various metrics like AUROC, PRR, and Accuracy.  
+- **Calibration**: Normalize and calibrate truth values for interpretable and comparable hallucination scores.  
+- **Integration**: Seamlessly works with Huggingface and LiteLLM.  
+- **Long-Form Generation**: Adapts detection methods to handle long-form text generations effectively.  
+- **Extendability**: Provides an intuitive interface for implementing new hallucination detection methods.  
+
+---
+
+## Installation  
+
+Install TruthTorchLM using pip:  
+
+```bash
+pip install TruthTorchLM
+```
+
+---
+
+## Quick Start  
+
+### Setting Up a Model  
+
+You can define your model and tokenizer using Huggingface or specify an API-based model:  
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import TruthTorchLM as ttlm
+import torch
+
+# Huggingface model
+model = AutoModelForCausalLM.from_pretrained(
+    "meta-llama/Llama-2-7b-chat-hf", 
+    torch_dtype=torch.bfloat16
+).to('cuda:0')
+tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf", use_fast=False)
+
+# API model
+api_model = "gpt-4o"
+```
+
+### Generating Text with Truth Values  
+
+TruthTorchLM generates messages with a truth value, indicating whether the model output is hallucinating or not. Various methods (called **truth methods**) can be used to detect hallucinations. Each method can have different algorithms and output ranges. Lower truth values generally suggest hallucinations. This functionality is mostly useful for short-form QA:
+```python
+# Define truth methods
+lars = ttlm.truth_methods.LARS()
+confidence = ttlm.truth_methods.Confidence()
+self_detection = ttlm.truth_methods.SelfDetection(number_of_questions=5)
+truth_methods = [lars, confidence, self_detection]
+```
+```python
+# Define a chat history
+chat = [{"role": "system", "content": "You are a helpful assistant. Give short and precise answers."},
+        {"role": "user", "content": "What is the capital city of France?"}]
+```
+```python
+# Generate text with truth values (Huggingface model)
+output_hf_model = ttlm.generate_with_truth_value(
+    model=model,
+    tokenizer=tokenizer,
+    messages=chat,
+    truth_methods=truth_methods,
+    max_new_tokens=100,
+    temperature=0.7
+)
+# Generate text with truth values (API model)
+output_api_model = ttlm.generate_with_truth_value(
+    model=api_model,
+    messages=chat,
+    truth_methods=truth_methods
+)
+```
+
+### Calibrating Truth Methods  
+Truth values for different methods may not be directly comparable. Use the `calibrate_truth_method` function to normalize truth values to a common range for better interpretability
+
+```python
+model_judge = ttlm.evaluators.ModelJudge('gpt-4o-mini')
+calibration_results = ttlm.calibrate_truth_method(
+    dataset='trivia_qa',
+    model=model,
+    truth_methods=truth_methods,
+    tokenizer=tokenizer,
+    correctness_evaluator=model_judge,
+    size_of_data=1000,
+    max_new_tokens=64
+)
+```
+
+### Evaluating Truth Methods  
+
+We can evaluate the truth methods with the `evaluate_truth_method` function. We can define different evaluation metrics including AUROC, AUPRC, AUARC, Accuracy, F1, Precision, Recall, PRR:  
+
+```python
+results = ttlm.evaluate_truth_method(
+    dataset='trivia_qa',
+    model=model,
+    truth_methods=truth_methods,
+    eval_metrics=['auroc', 'prr'],
+    tokenizer=tokenizer,
+    size_of_data=1000,
+    correctness_evaluator=model_judge,
+    max_new_tokens=64
+)
+```
+
+---
+
+## Available Hallucination Detection Methods  
+
+- **LARS**: [Do Not Design, Learn: A Trainable Scoring Function for Uncertainty Estimation in Generative LLMs](https://arxiv.org/pdf/2406.11278).  
+- **Confidence**: [Uncertainty Estimation in Autoregressive Structured Prediction](https://openreview.net/pdf?id=jN5y-zb5Q7m). 
+- **Entropy**:[Uncertainty Estimation in Autoregressive Structured Prediction](https://openreview.net/pdf?id=jN5y-zb5Q7m).  
+- **SelfDetection**: [Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Method](https://arxiv.org/pdf/2310.17918). 
+- **AttentionScore**: [LLM-Check: Investigating Detection of Hallucinations in Large Language Models](https://openreview.net/pdf?id=LYx4w3CAgy). 
+- **CrossExamination**: [LM vs LM: Detecting Factual Errors via Cross Examination](https://arxiv.org/pdf/2305.13281). 
+- **EccentricityConfidence**: [Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models](https://arxiv.org/pdf/2305.19187). 
+- **EccentricityUncertainty**: [Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models](https://arxiv.org/pdf/2305.19187).
+- **GoogleSearchCheck**: [FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios](https://arxiv.org/pdf/2307.13528). 
+- **Inside**: [INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection](https://openreview.net/pdf?id=Zj12nzlQbz). 
+- **KernelLanguageEntropy**: [Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities](https://arxiv.org/pdf/2405.20003). 
+- **MARS**: [MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMs](https://aclanthology.org/2024.acl-long.419.pdf). 
+- **MatrixDegreeConfidence**: [Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models](https://arxiv.org/pdf/2305.19187). 
+- **MatrixDegreeUncertainty**: [Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models](https://arxiv.org/pdf/2305.19187).
+- **MultiLLMCollab**: [Don’t Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration](https://arxiv.org/pdf/2402.00367). 
+- **NumSemanticSetUncertainty**: [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://arxiv.org/pdf/2302.09664). 
+- **PTrue**: [Language Models (Mostly) Know What They Know](https://arxiv.org/pdf/2207.05221). 
+- **Saplma**: [The Internal State of an LLM Knows When It’s Lying](https://aclanthology.org/2023.findings-emnlp.68.pdf). 
+- **SemanticEntropy**: [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://arxiv.org/pdf/2302.09664).
+- **sentSAR**: [Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models](https://aclanthology.org/2024.acl-long.276.pdf). 
+- **SumEigenUncertainty**: [Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models](https://arxiv.org/pdf/2305.19187).
+- **tokenSAR**: [Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models](https://aclanthology.org/2024.acl-long.276.pdf). 
+- **VerbalizedConfidence**: [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://openreview.net/pdf?id=g3faCfrwm7). 
+
+---
+
+## Contributors  
+
+- **Yavuz Faruk Bakman** (ybakman@usc.edu)  
+- **Duygu Nur Yaldiz** (yaldiz@usc.edu)  
+- **Sungmin Kang** (kangsung@usc.edu)  
+- **Hayrettin Eren Yildiz** (hayereyil@gmail.com)  
+- **Alperen Ozis** (alperenozis@gmail.com)  
+
+---
+
+## Citation  
+
+If you use TruthTorchLM in your research, please cite:  
+
+```bibtex
+@misc{truthtorchlm2025,
+  title={TruthTorchLM: A Comprehensive Library for Hallucination Detection in Large Language Models},
+  author={Yavuz Faruk Bakman, Duygu Nur Yaldiz,Sungmin Kang, Hayrettin Eren Yildiz, Alperen Ozis},
+  year={2025},
+  howpublished={GitHub},
+  url={https://github.com/Ybakman/TruthTorchLM}
+}
+```
+
+---
+
+## License  
+
+TruthTorchLM is released under the [MIT License](LICENSE).  
+
+For inquiries or support, feel free to contact the maintainers.
+

truthtorchlm-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

truthtorchlm-0.1.0/setup.py

@@ -0,0 +1,38 @@
+from setuptools import setup, find_packages
+
+requirements = ["aiohttp==3.9.3",
+    "evaluate==0.4.1",
+    "instructor",
+    "litellm==1.57.0",
+    "nest_asyncio==1.6.0", 
+    "numpy==1.26.4",
+    "outlines==0.1.4",
+    "pandas==2.2.3",
+    "pydantic",
+    "PyYAML",
+    "Requests==2.32.3",
+    "scikit_learn==1.4.1.post1",
+    "scipy==1.13.0",
+    "sentence_transformers==3.1.1",
+    "termcolor==2.5.0",
+    "torch==2.2.2", 
+    "tqdm==4.66.2",
+    "transformers==4.44.2",
+    "absl-py",
+    "nltk",
+    "rouge_score"]
+
+
+setup(
+    name="TruthTorchLM",  # Your package name
+    version="0.1.0",           # Package version
+    author="Yavuz Faruk Bakman",
+    author_email="ybakman@usc.edu",
+    description="TruthTorchLM is an open-source library designed to detect and mitigate hallucinations in text generation models. The library integrates state-of-the-art methods, offers comprehensive benchmarking tools across various tasks, and enables seamless integration with popular frameworks like Huggingface and LiteLLM.",
+    long_description=open("README.md").read(),
+    long_description_content_type="text/markdown",
+    package_dir={"": "src"},         # Maps the base package directory
+    packages=find_packages(where="src"),  # Automatically find and include all packages
+    install_requires=requirements,  # List of dependencies
+    python_requires=">=3.10",  # Minimum Python version
+)

truthtorchlm-0.1.0/src/TruthTorchLM/__init__.py

@@ -0,0 +1,16 @@
+from .truth_methods.truth_method import TruthMethod
+from TruthTorchLM import utils ##TODO do we really need to import this?
+from TruthTorchLM import scoring_methods
+from TruthTorchLM import truth_methods
+from .generation import generate_with_truth_value
+from .calibration import calibrate_truth_method
+from TruthTorchLM import evaluators
+from .evaluators import evaluate_truth_method
+from .templates import DEFAULT_USER_PROMPT, DEFAULT_SYSTEM_PROMPT ##TODO import all?
+from .availability import AVAILABLE_DATASETS, AVAILABLE_EVALUATION_METRICS
+from TruthTorchLM import normalizers
+
+from TruthTorchLM import long_form_generation
+
+
+#__all__ = ['generate_with_truth_value']

truthtorchlm-0.1.0/src/TruthTorchLM/availability.py

@@ -0,0 +1,14 @@
+AVAILABLE_API_MODELS = ['gpt-4o', 'gpt-4o-2024-05-13', 'gpt-4o-2024-08-06', 'chatgpt-4o-latest', 'gpt-4o-mini', 'gpt-4o-mini-2024-07-18', 
+                        'gpt-4-turbo','gpt-4-turbo-2024-04-09', 'gpt-4-turbo-preview', 'gpt-4-0125-preview', 'gpt-4-1106-preview', 'gpt-4', 
+                        'gpt-4-0613', 'gpt-4-0314', 'gpt-3.5-turbo-0125', 'gpt-3.5-turbo', 'gpt-3.5-turbo-1106', 'gpt-3.5-turbo-instruct', 'together_ai/togethercomputer/llama-2-70b'] 
+
+PROB_AVAILABLE_API_MODELS = ['gpt-4o', 'gpt-4o-2024-05-13', 'gpt-4o-2024-08-06', 'chatgpt-4o-latest', 'gpt-4o-mini', 'gpt-4o-mini-2024-07-18', 
+                        'gpt-4-turbo','gpt-4-turbo-2024-04-09', 'gpt-4-turbo-preview', 'gpt-4-0125-preview', 'gpt-4-1106-preview', 'gpt-4', 
+                        'gpt-4-0613', 'gpt-4-0314', 'gpt-3.5-turbo-0125', 'gpt-3.5-turbo', 'gpt-3.5-turbo-1106', 'gpt-3.5-turbo-instruct', 'together_ai/togethercomputer/llama-2-70b'] 
+
+ACTIVATION_AVAILABLE_API_MODELS = []
+
+AVAILABLE_DATASETS = ['trivia_qa', 'gsm8k', 'natural_qa', 'pop_qa', 'simple_qa']
+LONG_FORM_AVAILABLE_DATASETS = ['longfact_concepts', 'longfact_objects']
+
+AVAILABLE_EVALUATION_METRICS = ['auroc', 'auprc', 'auarc', 'accuracy', 'f1', 'precision', 'recall', 'prr']

truthtorchlm-0.1.0/src/TruthTorchLM/calibration.py

@@ -0,0 +1,36 @@
+from transformers import PreTrainedModel, PreTrainedTokenizer, PreTrainedTokenizerFast
+from typing import Union
+from TruthTorchLM.truth_methods import TruthMethod
+from TruthTorchLM.evaluators import CorrectnessEvaluator, ROUGE
+from TruthTorchLM.templates import DEFAULT_SYSTEM_BENCHMARK_PROMPT, DEFAULT_USER_PROMPT 
+from TruthTorchLM.utils.dataset_utils import get_dataset
+from TruthTorchLM.utils.eval_utils import run_over_dataset
+import numpy as np
+
+
+
+def calibrate_truth_method(dataset: Union[str, list], model:Union[str,PreTrainedModel],  truth_methods: list[TruthMethod], tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast] =None,
+correctness_evaluator:CorrectnessEvaluator = ROUGE(0.7), size_of_data:float = 1.0, previous_context:list =[{'role': 'system', 'content': DEFAULT_SYSTEM_BENCHMARK_PROMPT}], 
+ user_prompt:str = DEFAULT_USER_PROMPT, seed:int = 0, wandb_run = None, return_method_details:bool = False, wandb_push_method_details:bool = False, split = 'train', **kwargs):
+    
+    dataset = get_dataset(dataset, size_of_data=size_of_data, seed=seed, split = split)
+
+    output_dict = run_over_dataset(dataset, model, truth_methods, tokenizer = tokenizer, correctness_evaluator = correctness_evaluator, 
+                                   previous_context = previous_context, user_prompt = user_prompt, seed = seed, return_method_details = return_method_details, 
+                                   wandb_run = wandb_run, wandb_push_method_details = wandb_push_method_details, **kwargs)
+
+    for i, truth_method in enumerate(truth_methods):
+        truth_values = output_dict[f'truth_method_{i}']['truth_values']
+        truth_values = np.array(truth_values)
+        truth_values[np.isnan(truth_values)] = 0
+        correctness = output_dict['generation_correctness']
+        #if generation_correctness is -1, it means that the model didn't attempt to generate an answer, remove those from the evaluation
+        truth_method.normalizer.calibrate(generation_performance_scores=correctness, truth_values=truth_values)
+    return output_dict 
+
+
+    
+
+
+
+    

truthtorchlm-0.1.0/src/TruthTorchLM/evaluators/__init__.py

@@ -0,0 +1,8 @@
+from .correctness_evaluator import CorrectnessEvaluator
+from .rouge import ROUGE
+from .bleu import BLEU
+from .model_judge import ModelJudge
+from .eval_truth_method import evaluate_truth_method, get_metric_scores
+
+__all__ = ['CorrectnessEvaluator', 'ROUGE', 'BLEU', 'evaluate_truth_method', 'ModelJudge', 'get_metric_scores']
+

truthtorchlm-0.1.0/src/TruthTorchLM/evaluators/bleu.py

@@ -0,0 +1,20 @@
+
+from .correctness_evaluator import CorrectnessEvaluator
+import evaluate
+
+
+class BLEU(CorrectnessEvaluator):
+    def __init__(self, threshold: float = 0.5): 
+        super().__init__()
+        self.threshold = threshold  
+        self.bleu = evaluate.load('bleu')
+
+    def __call__(self, question_text:str, generated_text: str,  ground_truths: list[str], seed:int = None) -> bool:
+        for i in range(len(ground_truths)):
+            bleu_results = self.bleu.compute(predictions = [generated_text], references=[ground_truths[i]])
+            if bleu_results['bleu'] > self.threshold:
+                return 1
+        return 0
+    
+    def __str__(self):
+        return f"BLEU with threshold {self.threshold}"

truthtorchlm-0.1.0/src/TruthTorchLM/evaluators/correctness_evaluator.py

@@ -0,0 +1,14 @@
+from abc import ABC, abstractmethod
+
+class CorrectnessEvaluator(ABC):
+    
+    def __init__(self):
+        pass
+
+    @abstractmethod
+    def __call__(self, question_text:str, generated_text: str,  ground_truth_text: list[str], seed:int = None) -> int:
+        raise NotImplementedError("Subclasses must implement this method")
+    
+    @abstractmethod
+    def __str__(self):
+        raise NotImplementedError("Subclasses must implement this method")

truthtorchlm-0.1.0/src/TruthTorchLM/evaluators/eval_truth_method.py

@@ -0,0 +1,59 @@
+from transformers import PreTrainedModel, PreTrainedTokenizer, PreTrainedTokenizerFast
+from typing import Union
+from TruthTorchLM.truth_methods import TruthMethod
+from .correctness_evaluator import CorrectnessEvaluator 
+from .rouge import ROUGE
+from TruthTorchLM.availability import AVAILABLE_EVALUATION_METRICS
+from TruthTorchLM.templates import DEFAULT_SYSTEM_BENCHMARK_PROMPT, DEFAULT_USER_PROMPT
+from TruthTorchLM.utils.dataset_utils import get_dataset
+from TruthTorchLM.utils.eval_utils import metric_score, run_over_dataset
+import wandb
+
+
+def evaluate_truth_method(dataset: Union[str, list], model:Union[str,PreTrainedModel],  truth_methods: list[TruthMethod], tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast]=None, eval_metrics:list[str] = ['auroc'],
+                          correctness_evaluator:CorrectnessEvaluator = ROUGE(0.7), size_of_data = 1.0,  previous_context:list =[{'role': 'system', 'content': DEFAULT_SYSTEM_BENCHMARK_PROMPT}], 
+                          user_prompt:str = DEFAULT_USER_PROMPT, seed:int = 0, return_method_details:bool = False, wandb_run = None, wandb_push_method_details:bool = False, 
+                          batch_generation=True,  add_generation_prompt = True, continue_final_message = False, split='test',  **kwargs):
+    
+    dataset = get_dataset(dataset, size_of_data=size_of_data, seed=seed, split = split)
+
+    for eval_metric in eval_metrics:
+        if eval_metric not in AVAILABLE_EVALUATION_METRICS:
+            raise ValueError(f"Evaluation metric {eval_metric} is not available. Available evaluation metrics are: {AVAILABLE_EVALUATION_METRICS}")
+                
+    output_dict = run_over_dataset(dataset, model, truth_methods, tokenizer = tokenizer, correctness_evaluator = correctness_evaluator, 
+                                   previous_context = previous_context, user_prompt = user_prompt, seed = seed, return_method_details = return_method_details, 
+                                   wandb_run = wandb_run, wandb_push_method_details= wandb_push_method_details, 
+                                   batch_generation=batch_generation, add_generation_prompt=add_generation_prompt, continue_final_message=continue_final_message, **kwargs)
+
+    eval_list = get_metric_scores(output_dict=output_dict, eval_metrics=eval_metrics, seed=seed)
+
+    
+    if wandb_run:
+        wandb_run.log({'model_accuracy': sum(output_dict['generation_correctness'])/len(output_dict['generation_correctness'])})
+        
+        eval_dict = eval_list[0]
+        for key, _ in eval_dict.items():
+            methods = []
+            scores = []
+            for i, cur_eval_dict in enumerate(eval_list):
+                score = cur_eval_dict[key]
+                scores.append(score)
+                methods.append(str(truth_methods[i].__class__.__name__))
+                wandb_run.log({f'{key}_of_method_{i}_{str(truth_methods[i].__class__.__name__)}': score})
+
+            data = [[method, score] for (method, score) in zip(methods, scores)]
+            table = wandb.Table(data=data, columns = ["methods", "scores"])
+            wandb.log({f"{key}" : wandb.plot.bar(table, "methods", "scores",
+                            title=f"{key} Scores of Truth Methods")})  
+                
+    return {'eval_list': eval_list, 'output_dict': output_dict}
+
+
+def get_metric_scores(output_dict:dict, eval_metrics:list[str], seed:int=0):
+    truth_methods = output_dict['truth_methods']
+    eval_list = []
+    for i in range(len(truth_methods)):
+        eval_dict = metric_score(eval_metrics, output_dict['generation_correctness'], output_dict[f'truth_method_{i}']['truth_values'], output_dict[f'truth_method_{i}']['normalized_truth_values'], seed=seed)
+        eval_list.append(eval_dict)
+    return eval_list