tau-eval 0.1.0__tar.gz

tau_eval-0.1.0/PKG-INFO

@@ -0,0 +1,170 @@
+Metadata-Version: 2.4
+Name: tau-eval
+Version: 0.1.0
+Summary: Text Anonymization Evaluation Library
+Author-email: Gabriel Loiseau <gabriel.loiseau@hornetsecurity.com>
+Maintainer-email: Gabriel Loiseau <gabriel.loiseau@hornetsecurity.com>
+License-Expression: GPL-3.0
+Keywords: Text anonymization,evaluation,NLP
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: transformers<5.0.0,>=4.48.0
+Requires-Dist: sentence-transformers>=3.3.1
+Requires-Dist: torch>=2.5.0
+Requires-Dist: evaluate>=0.4.1
+Requires-Dist: datasets>=2.14.4
+Requires-Dist: huggingface-hub>=0.20.0
+Requires-Dist: tasknet>=1.57.0
+Requires-Dist: tasksource>=0.0.47
+Requires-Dist: ipywidgets>=8.1.5
+Requires-Dist: ipykernel>=6.29.5
+Requires-Dist: rich>=14.0.0
+Requires-Dist: accelerate>=1.6.0
+Requires-Dist: bert-score>=0.3.13
+Requires-Dist: nltk>=3.9.1
+Requires-Dist: rouge-score>=0.1.2
+Requires-Dist: faker>=37.1.0
+Requires-Dist: presidio-analyzer>=2.2.358
+Requires-Dist: presidio-anonymizer>=2.2.358
+Requires-Dist: pip>=25.0.1
+Requires-Dist: pytest>=8.3.5
+Provides-Extra: tests
+Requires-Dist: pytest; extra == "tests"
+Requires-Dist: accelerate>=0.20.3; extra == "tests"
+Provides-Extra: quality
+Requires-Dist: ruff; extra == "quality"
+Requires-Dist: pyyaml>=5.3.1; extra == "quality"
+
+# 𝜏 Tau-Eval: A Unified Evaluation Framework for Useful and Private Text Anonymization
+
+**Tau-Eval** is a user-friendly, modular, and customizable Python library designed to benchmark and evaluate text anonymization algorithms. It enables granular analysis of anonymization impacts from both privacy and utility perspectives. Tau-Eval seamlessly integrates with [LiteLLM](https://www.litellm.ai/) and [🤗 Hugging Face](https://huggingface.co/) to support a wide range of datasets, models, and evaluation metrics.
+
+<div align="center">
+
+[![GNU-GPLv3](https://img.shields.io/badge/license-%20%20GNU%20GPLv3%20-green?style=plastic)](LICENSE)
+[![v0.0.3](https://img.shields.io/badge/pypi-v0.0.3-orange)](https://pypi.org/project/xxx/xxx/)
+[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue)](https://www.python.org/downloads/)
+[![Tutorials](https://img.shields.io/badge/tutorials-colab-orange)](https://github.com/.../tree/main/examples)
+
+</div>
+
+
+## Installation
+
+### From PyPI
+
+Install Tau-Eval via pip:
+
+```
+pip install tau-eval
+```
+
+### From source
+
+To install from source:
+
+1) Clone this repository on your own path:
+```
+git clone https://github.com/gabrielloiseau/tau-eval.git
+cd tau-eval
+```
+
+2) Create an environment with your own preferred package manager. We used [python 3.9](https://www.python.org/downloads/release/python-390/) and dependencies listed in [`pyproject.toml`](pyproject.toml). If you use [conda](https://docs.conda.io/en/latest/), you can just run the following commands from the root of the project:
+
+```
+conda create --name taueval python=3.9         # create the environment
+conda activate taueval                         # activate the environment
+pip install --user -r pyproject.toml           # install the required packages
+```
+
+
+## Quickstart
+
+Tau-Eval is designed for flexibility. With just a few lines of code, you can set up and run evaluations.
+
+#### 1. Define Your Anonymization Model
+
+Create a custom anonymization model by extending the Anonymizer interface:
+```python
+from tau_eval.models import Anonymizer
+
+class TestModel(Anonymizer):
+    def __init__(self):
+        self.name = "Test Model"
+
+    def anonymize(self, text: str) -> str:
+        # Implement anonymization logic
+        return text
+
+    def anonymize_batch(self, texts: list[str]) -> list[str]:
+        # Batch processing
+        return texts
+
+```
+Or use prebuilt models from `tau_eval.models`.
+
+#### 2. Configure Evaluation Metrics
+Use built-in metrics from `tau_eval.metrics` or define your own following this signature:
+```python
+Callable[[str | list[str], str | list[str]], dict]
+```
+This allows complete control over what and how you evaluate.
+
+#### 3. Instantiate Tasks
+Tasks can be created using prebuilt options in `tau_eval.tasks`, or customized using `CustomTask`. Tau-Eval also supports [tasksource](https://github.com/sileod/tasksource) for dataset integration.
+
+#### 4. Configure and Run Your Experiment
+Define an experiment configuration:
+```python
+from tau_eval.config import ExperimentConfig
+
+config = ExperimentConfig(
+    exp_name="test-experiment",
+    classifier_name="medicalai/ClinicalBERT",
+    train_task_models=True,
+    train_with_generations=False,
+)
+```
+Run the experiment:
+```python
+from tau_eval.experiment import Experiment
+
+Experiment(
+    models=[m1, m2, m3, m4, m5],
+    metrics=["bertscore"],
+    tasks=[mednli, pii],
+    config=config
+).run()
+```
+#### 5. Visualize Results
+
+Tau-Eval includes built-in visualization tools to compare model anonymization strategies and evaluation results. You can find them with `tau_eval.visualization`. 
+
+
+## Tutorials
+
+You can find our tutorials to learn how to better leverage **Tau-Eval** in the [`examples/`]() folder.
+
+There you can also find a set of interesting case studies using real-world datasets! :chart_with_upwards_trend:
+
+
+
+## Contributors
+
+- **[Gabriel Loiseau](https://gabrielloiseau.github.io)**, *Hornetsecurty, Inria Lille*
+
+
+## Citation
+
+If you use 𝜏 **Tau-Eval** in your work, please cite our paper as follows:
+
+```
+
+
+```

tau_eval-0.1.0/README.md

@@ -0,0 +1,127 @@
+# 𝜏 Tau-Eval: A Unified Evaluation Framework for Useful and Private Text Anonymization
+
+**Tau-Eval** is a user-friendly, modular, and customizable Python library designed to benchmark and evaluate text anonymization algorithms. It enables granular analysis of anonymization impacts from both privacy and utility perspectives. Tau-Eval seamlessly integrates with [LiteLLM](https://www.litellm.ai/) and [🤗 Hugging Face](https://huggingface.co/) to support a wide range of datasets, models, and evaluation metrics.
+
+<div align="center">
+
+[![GNU-GPLv3](https://img.shields.io/badge/license-%20%20GNU%20GPLv3%20-green?style=plastic)](LICENSE)
+[![v0.0.3](https://img.shields.io/badge/pypi-v0.0.3-orange)](https://pypi.org/project/xxx/xxx/)
+[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue)](https://www.python.org/downloads/)
+[![Tutorials](https://img.shields.io/badge/tutorials-colab-orange)](https://github.com/.../tree/main/examples)
+
+</div>
+
+
+## Installation
+
+### From PyPI
+
+Install Tau-Eval via pip:
+
+```
+pip install tau-eval
+```
+
+### From source
+
+To install from source:
+
+1) Clone this repository on your own path:
+```
+git clone https://github.com/gabrielloiseau/tau-eval.git
+cd tau-eval
+```
+
+2) Create an environment with your own preferred package manager. We used [python 3.9](https://www.python.org/downloads/release/python-390/) and dependencies listed in [`pyproject.toml`](pyproject.toml). If you use [conda](https://docs.conda.io/en/latest/), you can just run the following commands from the root of the project:
+
+```
+conda create --name taueval python=3.9         # create the environment
+conda activate taueval                         # activate the environment
+pip install --user -r pyproject.toml           # install the required packages
+```
+
+
+## Quickstart
+
+Tau-Eval is designed for flexibility. With just a few lines of code, you can set up and run evaluations.
+
+#### 1. Define Your Anonymization Model
+
+Create a custom anonymization model by extending the Anonymizer interface:
+```python
+from tau_eval.models import Anonymizer
+
+class TestModel(Anonymizer):
+    def __init__(self):
+        self.name = "Test Model"
+
+    def anonymize(self, text: str) -> str:
+        # Implement anonymization logic
+        return text
+
+    def anonymize_batch(self, texts: list[str]) -> list[str]:
+        # Batch processing
+        return texts
+
+```
+Or use prebuilt models from `tau_eval.models`.
+
+#### 2. Configure Evaluation Metrics
+Use built-in metrics from `tau_eval.metrics` or define your own following this signature:
+```python
+Callable[[str | list[str], str | list[str]], dict]
+```
+This allows complete control over what and how you evaluate.
+
+#### 3. Instantiate Tasks
+Tasks can be created using prebuilt options in `tau_eval.tasks`, or customized using `CustomTask`. Tau-Eval also supports [tasksource](https://github.com/sileod/tasksource) for dataset integration.
+
+#### 4. Configure and Run Your Experiment
+Define an experiment configuration:
+```python
+from tau_eval.config import ExperimentConfig
+
+config = ExperimentConfig(
+    exp_name="test-experiment",
+    classifier_name="medicalai/ClinicalBERT",
+    train_task_models=True,
+    train_with_generations=False,
+)
+```
+Run the experiment:
+```python
+from tau_eval.experiment import Experiment
+
+Experiment(
+    models=[m1, m2, m3, m4, m5],
+    metrics=["bertscore"],
+    tasks=[mednli, pii],
+    config=config
+).run()
+```
+#### 5. Visualize Results
+
+Tau-Eval includes built-in visualization tools to compare model anonymization strategies and evaluation results. You can find them with `tau_eval.visualization`. 
+
+
+## Tutorials
+
+You can find our tutorials to learn how to better leverage **Tau-Eval** in the [`examples/`]() folder.
+
+There you can also find a set of interesting case studies using real-world datasets! :chart_with_upwards_trend:
+
+
+
+## Contributors
+
+- **[Gabriel Loiseau](https://gabrielloiseau.github.io)**, *Hornetsecurty, Inria Lille*
+
+
+## Citation
+
+If you use 𝜏 **Tau-Eval** in your work, please cite our paper as follows:
+
+```
+
+
+```

tau_eval-0.1.0/pyproject.toml

@@ -0,0 +1,109 @@
+[project]
+name = "tau-eval"
+version = "0.1.0"
+description = "Text Anonymization Evaluation Library"
+license = "GPL-3.0"
+readme = "README.md"
+authors = [
+    { name = "Gabriel Loiseau", email = "gabriel.loiseau@hornetsecurity.com" },
+]
+maintainers = [
+    { name = "Gabriel Loiseau", email = "gabriel.loiseau@hornetsecurity.com" },
+]
+requires-python = ">=3.9"
+keywords = [
+    "Text anonymization",
+    "evaluation",
+    "NLP",
+]
+classifiers = [
+    "Intended Audience :: Science/Research",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+    "transformers>=4.48.0,<5.0.0",
+    "sentence-transformers>=3.3.1",
+    "torch>=2.5.0",
+    "evaluate>=0.4.1",
+    "datasets>=2.14.4",
+    "huggingface-hub>=0.20.0",
+    "tasknet>=1.57.0",
+    "tasksource>=0.0.47",
+    "ipywidgets>=8.1.5",
+    "ipykernel>=6.29.5",
+    "rich>=14.0.0",
+    "accelerate>=1.6.0",
+    "bert-score>=0.3.13",
+    "nltk>=3.9.1",
+    "rouge-score>=0.1.2",
+    "faker>=37.1.0",
+    "presidio-analyzer>=2.2.358",
+    "presidio-anonymizer>=2.2.358",
+    "pip>=25.0.1",
+    "pytest>=8.3.5",
+]
+
+
+[project.optional-dependencies]
+tests = [
+    "pytest",
+    "accelerate>=0.20.3",
+]
+quality = [ 
+    "ruff",
+    "pyyaml>=5.3.1",
+]
+
+
+[tool.ruff]
+line-length = 119
+fix = true
+
+[tool.ruff.lint]
+# Never enforce `E501` (line length violations).
+ignore = ["C901", "E501", "E741", "F402", "F823", "C416" ]
+select = ["C", "E", "F", "I", "W"]
+
+[tool.ruff.lint.per-file-ignores]
+"examples/**" = [
+    # Ignore `E402` (import violations) in all examples
+    "E402", 
+    # Ignore missing required imports
+    "I002"
+    ]
+"docs/**" = [
+    # Ignore missing required imports
+    "I002"
+    ]
+
+[tool.ruff.lint.isort]
+lines-after-imports = 2
+known-first-party = ["transformers"]
+
+[tool.ruff.format]
+# Like Black, use double quotes for strings.
+quote-style = "double"
+
+# Like Black, indent with spaces, rather than tabs.
+indent-style = "space"
+
+# Like Black, respect magic trailing commas.
+skip-magic-trailing-comma = false
+
+# Like Black, automatically detect the appropriate line ending.
+line-ending = "auto"
+
+
+[tool.pytest.ini_options]
+testpaths = [
+    "tests"
+]
+addopts = "--strict-markers -m 'not slow'"
+markers = [
+    "slow: marks tests as slow"
+]
+

tau_eval-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

tau_eval-0.1.0/tau_eval/__init__.py

@@ -0,0 +1,18 @@
+"""Top-level package for tau-eval."""
+
+from . import (
+    logger,
+    models,
+    utils,
+)
+from .experiment import Experiment, ExperimentConfig
+
+
+__all__ = [
+    "Experiment",
+    "ExperimentConfig",
+    "models",
+    "utils",
+    "logger",
+]
+__version__ = "0.1.0"

tau_eval-0.1.0/tau_eval/experiment.py

@@ -0,0 +1,231 @@
+import json
+import os
+from dataclasses import dataclass, field
+
+import pandas as pd
+from rich.table import Table
+from tasknet import Task
+
+from tau_eval.tasks.customtask import CustomTask
+
+from .logger import logger
+from .models import Anonymizer
+from .utils import run_models_on_custom_task, run_models_on_task
+
+
+def rich_display_dataframe(df, title="Dataframe") -> None:
+    """Display dataframe as table using rich library.
+    Args:
+        df (pd.DataFrame): dataframe to display
+        title (str, optional): title of the table. Defaults to "Dataframe".
+    Raises:
+        NotRenderableError: if dataframe cannot be rendered
+    Returns:
+        rich.table.Table: rich table
+    """
+    from rich import print
+
+    # ensure dataframe contains only string values
+    df = df.astype(str)
+
+    table = Table(title=title)
+    for col in df.columns:
+        table.add_column(col)
+    for row in df.values:
+        table.add_row(*row)
+    print(table)
+
+@dataclass
+class ExperimentConfig:
+    """Evaluation experiment config"""
+
+    exp_name: str = os.path.basename(__file__)[: -len(".py")]
+    classifier_name: str = "answerdotai/ModernBERT-base"
+    train_task_models: bool = False
+    train_with_generations: bool = False
+    device: str | None = "cuda"
+    classifier_args: dict = field(default_factory=dict)
+
+
+class Experiment:
+    def __init__(
+        self,
+        models: list[Anonymizer],
+        metrics: list[str],
+        tasks: list[Task],
+        config: ExperimentConfig = ExperimentConfig(),
+    ):
+        self.models = models
+        self.metrics = metrics
+        self.tasks = tasks
+        self.args = config
+
+    def run(self, output_dir="", device="cuda"):
+        self.output_dir = output_dir
+        logger.info("Running experiment...")
+        out = {}
+
+        for i, task in enumerate(self.tasks):
+            logger.info(f"Running task: {i}")
+            if isinstance(task, CustomTask):
+                results = run_models_on_custom_task(self.models, task, self.metrics)
+            else:
+                results = run_models_on_task(
+                    self.models,
+                    task,
+                    self.metrics,
+                    self.args.classifier_name,
+                    self.args.train_task_models,
+                    self.args.train_with_generations,
+                    device,
+                )
+            if hasattr(task, "name"):
+                out[f"{task.name}_{i}"] = results
+            else:
+                out[f"{task.__class__.__name__}_{i}"] = results
+
+        self.results = out
+        with open(self.output_dir, "w") as f:
+            json.dump(out, f)
+            logger.info("Results saved")
+
+    def summary(self, output_dir=None, to_rich=False):
+        if output_dir is not None:
+            results = json.load(open(output_dir))
+        else:
+            results = json.load(open(self.output_dir))
+
+        task_dataframes = {}
+
+        for task_name, task in results.items():
+            # Prepare columns for DataFrame
+            if isinstance(task, CustomTask):
+                columns = ["Model Name"] + task["metrics"]
+                rows = []
+
+                # Add original metrics
+                original_metrics = task.get("original_metrics", {})
+                if original_metrics:
+                    original_row = ["Original"]
+                    for m in task["metrics"]:
+                        if m == "cola":
+                            original_row.append(round(original_metrics.get("cola", 0), 4))
+                        else:
+                            original_row.append("-")
+                    rows.append(original_row)
+                    # Add rewritten metrics
+                for model_name, model in task.items():
+                    if model_name in ["original_metrics", "metrics"]:
+                        continue
+
+                    row = [model_name]
+
+                    for m in task["metrics"]:
+                        row.append(round(model.get(m, 0), 4))
+
+                    rows.append(row)
+
+            else:
+                columns = ["Model Name", "Accuracy", "F1"] + task["metrics"]
+                rows = []
+
+                # Add original metrics
+                original_metrics = task.get("original_metrics", {})
+                if original_metrics:
+                    original_row = ["Original"]
+                    original_row.append(round(original_metrics.get("test_accuracy", 0), 4))
+                    original_row.append(round(original_metrics.get("test_f1", 0), 4))
+                    for m in task["metrics"]:
+                        if m == "cola":
+                            original_row.append(round(original_metrics.get("cola", 0), 4))
+                        else:
+                            original_row.append("-")
+                    rows.append(original_row)
+
+                # Add rewritten metrics
+                for model_name, model in task.items():
+                    if model_name in ["original_metrics", "metrics"]:
+                        continue
+
+                    row = [model_name]
+                    row.append(model.get("test_accuracy", 0))
+                    row.append(model.get("test_f1", 0))
+
+                    for m in task["metrics"]:
+                        row.append(round(model.get(m, 0), 4))
+
+                    rows.append(row)
+
+            # Create DataFrame for the task
+            task_dataframe = pd.DataFrame(rows, columns=columns)
+            task_dataframes[task_name] = task_dataframe
+        if to_rich:
+            for t, value in task_dataframes.items():
+                rich_display_dataframe(value)
+
+        return task_dataframes
+
+    def plot_tradeoff(
+        self,
+        task_name1,
+        target1,
+        task_name2,
+        target2,
+        x_label="Metric 1",
+        y_label="Metric 2",
+        title="Tradeoff Between Metrics",
+        figsize=(4, 4),
+    ):
+        from itertools import cycle
+
+        import matplotlib.pyplot as plt
+
+        task1 = self.results[task_name1]
+        task2 = self.results[task_name2]
+        model_names = []
+        x_vals = []
+        y_vals = []
+
+        for model_name, model in task1.items():
+            model_names.append(model_name)
+            if model_name in ["original_metrics", "metrics"]:
+                continue
+            else:
+                x_vals.append(model.get(target1))
+
+        for model_name, model in task2.items():
+            model_names.append(model_name)
+            if model_name in ["original_metrics", "metrics"]:
+                continue
+            else:
+                y_vals.append(model.get(target2))
+
+        markers = cycle(
+            ["o", "s", "D", "^", "v", "<", ">", "p", "*", "h", "H"]
+        )  # Cycle through different marker styles
+        plt.figure(figsize=figsize)
+
+        for i, (x_val, y_val) in enumerate(zip(x_vals, y_vals)):
+            marker = next(markers)
+            plt.scatter(
+                x_val,
+                y_val,
+                label=model_names[i] if model_names else f"Model {i + 1}",
+                marker=marker,
+                s=100,
+                alpha=0.7,
+                edgecolor="k",
+            )
+            # Annotate points with model names if provided
+            if model_names:
+                plt.text(x_val, y_val, model_names[i], fontsize=9, ha="right", va="bottom")
+
+        plt.axhline(0, color="gray", linewidth=0.5, linestyle="--")
+        plt.axvline(0, color="gray", linewidth=0.5, linestyle="--")
+        plt.grid(alpha=0.3)
+
+        plt.xlabel(x_label)
+        plt.ylabel(y_label)
+        plt.title(title)
+        plt.legend(loc="best", fontsize=9)
+        plt.tight_layout()

tau_eval-0.1.0/tau_eval/logger.py

@@ -0,0 +1,28 @@
+import logging
+
+from rich.logging import RichHandler
+
+
+logger = logging.getLogger(__name__)
+
+# the handler determines where the logs go: stdout/file
+shell_handler = RichHandler()
+file_handler = logging.FileHandler("debug.log")
+
+logger.setLevel(logging.DEBUG)
+shell_handler.setLevel(logging.DEBUG)
+file_handler.setLevel(logging.DEBUG)
+
+# the formatter determines what our logs will look like
+fmt_shell = "%(message)s"
+fmt_file = "%(levelname)s %(asctime)s [%(filename)s:%(funcName)s:%(lineno)d] %(message)s"
+
+shell_formatter = logging.Formatter(fmt_shell)
+file_formatter = logging.Formatter(fmt_file)
+
+# here we hook everything together
+shell_handler.setFormatter(shell_formatter)
+file_handler.setFormatter(file_formatter)
+
+logger.addHandler(shell_handler)
+logger.addHandler(file_handler)