modern-bert-score 0.0.1__tar.gz

modern_bert_score-0.0.1/LICENSE.md

@@ -0,0 +1,13 @@
+Copyright 2026 Philipp Koch
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

modern_bert_score-0.0.1/PKG-INFO

@@ -0,0 +1,124 @@
+Metadata-Version: 2.4
+Name: modern-bert-score
+Version: 0.0.1
+Summary: A reimplementation of the BERTScore metric optimized for modern inference workflows.
+Author-email: Philipp Koch <PhillKoch@protonmail.com>
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/pypa/sampleproject
+Project-URL: Issues, https://github.com/pypa/sampleproject/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Environment :: GPU :: NVIDIA CUDA
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE.md
+Requires-Dist: sentence-transformers>=5.2.3
+Requires-Dist: torch>=2.9.1
+Requires-Dist: transformers>=4.57.6
+Provides-Extra: vllm
+Requires-Dist: vllm>=0.16.0; extra == "vllm"
+Provides-Extra: dev
+Requires-Dist: sphinx; extra == "dev"
+Requires-Dist: sphinx_rtd_theme; extra == "dev"
+Requires-Dist: myst-parser; extra == "dev"
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: flake8; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+Requires-Dist: flake8; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Dynamic: license-file
+
+# Modern BERTScore for Fast Inference
+
+[![CI](https://github.com/LazerLambda/modern-bert-score/actions/workflows/ci.yml/badge.svg)](https://github.com/LazerLambda/modern-bert-score/actions/workflows/ci.yml)
+[![Python 3.12](https://img.shields.io/badge/python-3.12-blue.svg)](https://www.python.org/downloads/release/python-3120/)
+[![Python 3.13](https://img.shields.io/badge/python-3.13-blue.svg)](https://www.python.org/downloads/release/python-3130/)
+[![Python 3.14](https://img.shields.io/badge/python-3.14-blue.svg)](https://www.python.org/downloads/)
+[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+
+![Performance](runtime_comp_nvidia_rtx_4060.png)
+
+**Modern-BERT-Score** is a reimplementation of the BERTScore metric introduced by [Zhang et al., 2019](https://arxiv.org/abs/1904.09675), optimized for modern inference workflows using [SentenceTransformers](https://www.sbert.net/) and [vLLM](https://vllm.ai/).  
+
+This library provides fast, GPU-accelerated scoring for text generation evaluation, making BERTScore practical for large-scale inference tasks.
+
+---
+
+## ⚡ Features
+- Fast, efficient computation with optional vLLM support  
+- Compatible with all Hugging Face transformer models  
+- Supports truncated and optimized model versions for faster inference  
+- Works seamlessly with both CPU and GPU setups  
+
+---
+
+## 📦 Installation
+
+Modern-BERT-Score comes in **two variants**: a base version and a vLLM-enhanced version. For vLLM, an NVIDIA GPU is strongly recommended.  
+
+### Base Version
+```bash
+pip install modern-bert-score
+```
+
+### vLLM Version
+```{bash}
+pip install modern-bert-score[vllm]
+```
+This implementation is significantly faster than the original BERTScore, especially with GPU acceleration.
+
+## 📝 BERTScore
+BERTScore ([Zhang et al., 2019](https://arxiv.org/abs/1904.09675)) evaluates the similarity between candidate and reference texts by comparing their contextual embeddings from a pre-trained transformer model. For each token in the candidate, it finds the most similar token in the reference (using cosine similarity) and aggregates these scores to compute **precision**, **recall**, and **F1**. Optionally, **IDF-weighting** can be applied to give more importance to rare and informative words, improving the metric’s sensitivity to meaningful content over common words. Additionally, optional **Baseline Rescaling** shifts the scores such that the score is in the range of [0,1]. This approach captures semantic similarity beyond exact word matches, making it robust for tasks such as machine translation and text generation evaluation.  
+
+The following figure, taken from the original paper, illustrates how BERTScore works:
+
+![BERTScore](zhang_19_figure_1.png)
+
+## 🛠 Usage
+### Example
+```python
+from modern_bert_score import BertScore
+
+candidates = ["Hello World!", "A robin is a bird."]
+references = ["Hi World!", "A robin is not a bird."]
+
+metric = BertScore(model_id="roberta-base")
+scores = metric(candidates, references)
+
+# scores is a list of (Precision, Recall, F1) tuples
+# To get separate lists of P, R, F1:
+P, R, F1 = zip(*scores)
+
+print("Precision scores:", P)
+print("Recall scores:", R)
+print("F1 scores:", F1)
+```
+
+## ⚠️ NOTICE
+
+- For best performance, an optimal layer should be used for each model.  
+- To find the optimal layer, [please use this script from the original BERTScore implementation](https://github.com/Tiiiger/bert_score/tree/master/tune_layers).  
+
+Some pre-truncated models optimized for vLLM are available on [Hugging Face](https://huggingface.co/collections/LazerLambda/modern-bertscore) and directly available in this library:
+
+- `LazerLambda/ModernBERT-base-ModBERTScore-12` -> `ModernBERTBaseScore`
+- `LazerLambda/ModernBERT-large-ModBERTScore-19` -> `ModernBERTLargeScore` 
+- `LazerLambda/roberta-base-ModBERTScore-10`  -> `RobertaBaseScore`
+- `LazerLambda/roberta-large-ModBERTScore-17` -> `RobertaLargeScore`
+- `LazerLambda/roberta-large-mnli-ModBERTScore-19` -> `RobertaLargeMNLIScore`
+
+
+## 🗺 Roadmap
+
+- [x] Implement base version and vLLM addon  
+- [x] Add IDF-weighted scoring  
+- [ ] Add baseline-rescaling and scripts for identifying optimal baselines  
+- [ ] Add model (vLLM-)adaptation script for slicing the model
+- [ ] Add multilingual support
+- [ ] Add CLI tool

modern_bert_score-0.0.1/README.md

@@ -0,0 +1,88 @@
+# Modern BERTScore for Fast Inference
+
+[![CI](https://github.com/LazerLambda/modern-bert-score/actions/workflows/ci.yml/badge.svg)](https://github.com/LazerLambda/modern-bert-score/actions/workflows/ci.yml)
+[![Python 3.12](https://img.shields.io/badge/python-3.12-blue.svg)](https://www.python.org/downloads/release/python-3120/)
+[![Python 3.13](https://img.shields.io/badge/python-3.13-blue.svg)](https://www.python.org/downloads/release/python-3130/)
+[![Python 3.14](https://img.shields.io/badge/python-3.14-blue.svg)](https://www.python.org/downloads/)
+[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+
+![Performance](runtime_comp_nvidia_rtx_4060.png)
+
+**Modern-BERT-Score** is a reimplementation of the BERTScore metric introduced by [Zhang et al., 2019](https://arxiv.org/abs/1904.09675), optimized for modern inference workflows using [SentenceTransformers](https://www.sbert.net/) and [vLLM](https://vllm.ai/).  
+
+This library provides fast, GPU-accelerated scoring for text generation evaluation, making BERTScore practical for large-scale inference tasks.
+
+---
+
+## ⚡ Features
+- Fast, efficient computation with optional vLLM support  
+- Compatible with all Hugging Face transformer models  
+- Supports truncated and optimized model versions for faster inference  
+- Works seamlessly with both CPU and GPU setups  
+
+---
+
+## 📦 Installation
+
+Modern-BERT-Score comes in **two variants**: a base version and a vLLM-enhanced version. For vLLM, an NVIDIA GPU is strongly recommended.  
+
+### Base Version
+```bash
+pip install modern-bert-score
+```
+
+### vLLM Version
+```{bash}
+pip install modern-bert-score[vllm]
+```
+This implementation is significantly faster than the original BERTScore, especially with GPU acceleration.
+
+## 📝 BERTScore
+BERTScore ([Zhang et al., 2019](https://arxiv.org/abs/1904.09675)) evaluates the similarity between candidate and reference texts by comparing their contextual embeddings from a pre-trained transformer model. For each token in the candidate, it finds the most similar token in the reference (using cosine similarity) and aggregates these scores to compute **precision**, **recall**, and **F1**. Optionally, **IDF-weighting** can be applied to give more importance to rare and informative words, improving the metric’s sensitivity to meaningful content over common words. Additionally, optional **Baseline Rescaling** shifts the scores such that the score is in the range of [0,1]. This approach captures semantic similarity beyond exact word matches, making it robust for tasks such as machine translation and text generation evaluation.  
+
+The following figure, taken from the original paper, illustrates how BERTScore works:
+
+![BERTScore](zhang_19_figure_1.png)
+
+## 🛠 Usage
+### Example
+```python
+from modern_bert_score import BertScore
+
+candidates = ["Hello World!", "A robin is a bird."]
+references = ["Hi World!", "A robin is not a bird."]
+
+metric = BertScore(model_id="roberta-base")
+scores = metric(candidates, references)
+
+# scores is a list of (Precision, Recall, F1) tuples
+# To get separate lists of P, R, F1:
+P, R, F1 = zip(*scores)
+
+print("Precision scores:", P)
+print("Recall scores:", R)
+print("F1 scores:", F1)
+```
+
+## ⚠️ NOTICE
+
+- For best performance, an optimal layer should be used for each model.  
+- To find the optimal layer, [please use this script from the original BERTScore implementation](https://github.com/Tiiiger/bert_score/tree/master/tune_layers).  
+
+Some pre-truncated models optimized for vLLM are available on [Hugging Face](https://huggingface.co/collections/LazerLambda/modern-bertscore) and directly available in this library:
+
+- `LazerLambda/ModernBERT-base-ModBERTScore-12` -> `ModernBERTBaseScore`
+- `LazerLambda/ModernBERT-large-ModBERTScore-19` -> `ModernBERTLargeScore` 
+- `LazerLambda/roberta-base-ModBERTScore-10`  -> `RobertaBaseScore`
+- `LazerLambda/roberta-large-ModBERTScore-17` -> `RobertaLargeScore`
+- `LazerLambda/roberta-large-mnli-ModBERTScore-19` -> `RobertaLargeMNLIScore`
+
+
+## 🗺 Roadmap
+
+- [x] Implement base version and vLLM addon  
+- [x] Add IDF-weighted scoring  
+- [ ] Add baseline-rescaling and scripts for identifying optimal baselines  
+- [ ] Add model (vLLM-)adaptation script for slicing the model
+- [ ] Add multilingual support
+- [ ] Add CLI tool

modern_bert_score-0.0.1/modern_bert_score/__init__.py

@@ -0,0 +1,17 @@
+from .bert_score import (
+    BertScore,
+    ModernBERTBaseScore,
+    ModernBERTLargeScore,
+    RobertaBaseScore,
+    RobertaLargeMNLIScore,
+    RobertaLargeScore,
+)
+
+__all__ = [
+    "BertScore",
+    "ModernBERTBaseScore",
+    "ModernBERTLargeScore",
+    "RobertaBaseScore",
+    "RobertaLargeScore",
+    "RobertaLargeMNLIScore",
+]

modern_bert_score-0.0.1/modern_bert_score/bert_score.py

@@ -0,0 +1,310 @@
+from collections import Counter, defaultdict
+from functools import partial, reduce
+from itertools import chain, islice
+from math import log
+from multiprocessing import Pool
+from typing import Any, Dict, Generator, List, Optional, Tuple, Union
+
+import torch
+from transformers import AutoTokenizer
+from transformers.tokenization_utils_fast import PreTrainedTokenizerFast
+
+from modern_bert_score.consts import BASELINES
+from modern_bert_score.inference import STInference, VLLMInference
+
+
+class BertScore:
+    inference_engine: Optional[Union[STInference, VLLMInference]]
+
+    def __init__(
+        self,
+        model_id: str = "answerdotai/ModernBERT-base",
+        idf_weighting: bool = False,
+        baseline_rescaling: bool = False,
+        custom_baseline: Optional[Tuple[float, float, float]] = None,
+        device: str = "cpu",  # TODO Enum cuda, mlx, cpu?
+        backend: str = "default",  # TODO Enum default, vllm, onnx, etc
+        sentence_transformers_args: Optional[Dict[str, Any]] = None,
+        vllm_args: Optional[Dict[str, Any]] = None,
+    ):
+        self.tokenizer: PreTrainedTokenizerFast = AutoTokenizer.from_pretrained(model_id)
+        self.idf_weighting: bool = idf_weighting
+        if baseline_rescaling:
+            if model_id in BASELINES.keys():
+                self.baseline: Tuple[float, float, float] = BASELINES[model_id]
+                if isinstance(self.baseline, tuple):
+                    self.baseline = self.baseline
+            elif custom_baseline is not None:
+                self.baseline = custom_baseline
+            else:
+                raise ValueError(
+                    (
+                        "Baseline rescaling enabled but no"
+                        f"baseline found for model {model_id}"
+                        " and no custom baseline provided."
+                    )
+                )
+        self.baseline_rescaling: bool = baseline_rescaling
+        if backend == "vllm":
+            self.inference_engine = VLLMInference(
+                model=model_id,
+                runner="pooling",
+                convert="embed",
+                **(vllm_args or {}),
+            )
+        elif backend == "default":
+            self.inference_engine = STInference(
+                model_id, device=device, **(sentence_transformers_args or {})
+            )
+        else:
+            raise ValueError(f"Unsupported backend {backend}")
+
+    def __call__(
+        self,
+        candidates: Union[str, List[str]],
+        references: Union[str, List[str]],
+        **kwargs: Any
+    ) -> List[Tuple[float, float, float]]:
+        assert reduce(
+            lambda acc, x: acc and isinstance(x, str), candidates, True
+        ), "Candidates must be a list of strings or a string."
+        assert reduce(
+            lambda acc, x: acc and isinstance(x, str), references, True
+        ), "References must be a list of strings or a string."
+        if isinstance(candidates, str):
+            candidates = [candidates]
+        if isinstance(references, str):
+            references = [references]
+        if len(candidates) != len(references):
+            raise ValueError(
+                "Number of candidates and references must be the same."
+            )
+        if len(candidates) == 0:
+            return []
+        candidates = [c.strip() for c in candidates]
+        references = [r.strip() for r in references]
+        if self.idf_weighting:
+            idf_dict_ref, input_ids_ref = self.get_idf_dict(references)
+            input_ids_cand = self._tokenize_data(candidates)
+        else:
+            idf_dict_ref = None
+
+        if self.inference_engine is None:
+            raise ValueError(
+                "Inference engine not initialized. Check backend "
+                "configuration."
+            )
+        cand_embs, ref_embs = self.inference_engine.inference(
+            candidates, references, **kwargs
+        )
+        if self.idf_weighting:
+            scores = [
+                self.bert_score(
+                    candidates=c,
+                    references=r,
+                    idf_dict_ref=idf_dict_ref,
+                    input_ids_cand=ids_cand,
+                    input_ids_ref=ids_ref,
+                )
+                for c, r, ids_cand, ids_ref in zip(
+                    cand_embs, ref_embs, input_ids_cand, input_ids_ref
+                )
+            ]
+        else:
+            scores = [
+                self.bert_score(candidates=c, references=r)
+                for c, r in zip(cand_embs, ref_embs)
+            ]
+        if self.baseline_rescaling:
+            rescaled_scores = []
+            for p, r, f1 in scores:
+                rescaled_p = (p - self.baseline[0]) / (1 - self.baseline[0])
+                rescaled_r = (r - self.baseline[1]) / (1 - self.baseline[1])
+                rescaled_f1 = (f1 - self.baseline[2]) / (1 - self.baseline[2])
+                rescaled_scores.append((rescaled_p, rescaled_r, rescaled_f1))
+            return rescaled_scores
+        return scores
+
+    @staticmethod
+    def _check_nan(f1: torch.Tensor) -> torch.Tensor:
+        if torch.isnan(f1):
+            f1 = torch.Tensor([0.0])
+        return f1
+
+
+    def bert_score(
+        self,
+        candidates: torch.Tensor,
+        references: torch.Tensor,
+        idf_dict_ref: Optional[Dict[int, float]] = None,
+        input_ids_cand: Optional[List[int]] = None,
+        input_ids_ref: Optional[List[int]] = None,
+    ) -> Tuple[float, float, float]:
+        has_idf_dict = idf_dict_ref is not None
+        has_input_ids = (
+            input_ids_cand is not None and input_ids_ref is not None
+        )
+        if has_idf_dict != has_input_ids:
+            raise ValueError(
+                "`idf_dict` and `input_ids` must either both be provided or "
+                "both be None."
+            )
+        # TODO: w cuda?
+        assert len(candidates.shape) == 2 and len(references.shape) == 2
+        candidates = candidates[1:-1]  # remove CLS and SEP
+        references = references[1:-1]
+        similarities: torch.Tensor = candidates @ references.T
+        r_bert = similarities.max(dim=0).values.cpu()
+        p_bert = similarities.max(dim=1).values.cpu()
+        if idf_dict_ref and input_ids_cand and input_ids_ref:
+            idf_weights_cand = torch.tensor(
+                [idf_dict_ref[tok_id] for tok_id in input_ids_cand]
+            )
+            idf_weights_ref = torch.tensor(
+                [idf_dict_ref[tok_id] for tok_id in input_ids_ref]
+            )
+            idf_weights_cand /= idf_weights_cand.sum()
+            idf_weights_ref /= idf_weights_ref.sum()
+            p_bert = (p_bert * idf_weights_cand).sum()
+            r_bert = (r_bert * idf_weights_ref).sum()
+        else:
+            r_bert = r_bert.mean()
+            p_bert = p_bert.mean()
+        f1 = 2 * p_bert * r_bert / (p_bert + r_bert)
+        # handle p_bert + r_bert == 0
+        f1 = self._check_nan(f1)
+        return p_bert.item(), r_bert.item(), f1.item()
+
+    @staticmethod
+    def _batchify(
+        iterable: List[str], batch_size: int
+    ) -> Generator[List[str], None, None]:
+        iterator = iter(iterable)
+        while True:
+            batch = list(islice(iterator, batch_size))
+            if not batch:
+                break
+            yield batch
+
+    @staticmethod
+    def _process_batch(
+        batch: List[str],
+        tokenizer: PreTrainedTokenizerFast,
+        ignore_counter: bool = False,
+    ) -> Tuple[Counter[int], List[List[int]]]:
+        stripped_batch = [sample.strip() for sample in batch]
+
+        encoded_batch = tokenizer(
+            stripped_batch,
+            add_special_tokens=True,
+            truncation=True,
+            return_attention_mask=False,
+            return_token_type_ids=False,
+        )["input_ids"]
+        encoded_batch = [e[1:-1] for e in encoded_batch]  # remove CLS and SEP
+        if ignore_counter:
+            return Counter(), encoded_batch
+        else:
+            batch_count: Counter[int] = Counter(
+                chain.from_iterable(map(set, encoded_batch))
+            )
+            return batch_count, encoded_batch
+
+    def _tokenize_data(
+            self,
+            corpus: List[str],
+            batch_size: int = 100_000,
+            nthreads: int = 4) -> List[List[int]]:
+        collected_input_ids: List[List[int]] = []
+
+        process_partial = partial(
+            self._process_batch, tokenizer=self.tokenizer, ignore_counter=True
+        )
+        batches = self._batchify(corpus, batch_size)
+
+        if nthreads > 0:
+            with Pool(nthreads) as pool:
+                for batch_result in pool.imap(
+                    process_partial, batches, chunksize=1
+                ):
+                    _, batch_input_ids = batch_result
+                    collected_input_ids.extend(batch_input_ids)
+        else:
+            for batch_result in map(process_partial, batches):
+                _, batch_input_ids = batch_result
+                collected_input_ids.extend(batch_input_ids)
+        return collected_input_ids
+
+    def get_idf_dict(
+        self,
+        corpus: List[str],
+        nthreads: int = 4,
+        batch_size: int = 100_000,
+    ) -> Tuple[Dict[int, float], List[List[int]]]:  # TODO: Return dict
+        """Build an IDF (Inverse-Document-Frequency) dictionary for a corpus.
+
+        When ``return_input_ids`` is true, this also returns the tokenized
+        ``input_ids`` for each corpus entry in the same order as ``corpus``.
+        """
+        idf_count: Counter[int] = Counter()
+        collected_input_ids: List[List[int]] = []
+        num_docs = len(corpus)
+
+        process_partial = partial(
+            self._process_batch, tokenizer=self.tokenizer
+        )
+        batches = self._batchify(corpus, batch_size)
+
+        if nthreads > 0:
+            with Pool(nthreads) as pool:
+                for batch_result in pool.imap(
+                    process_partial, batches, chunksize=1
+                ):
+                    batch_count, batch_input_ids = batch_result
+                    collected_input_ids.extend(batch_input_ids)
+                    idf_count.update(batch_count)
+        else:
+            for batch_result in map(process_partial, batches):
+                batch_count, batch_input_ids = batch_result
+                collected_input_ids.extend(batch_input_ids)
+                idf_count.update(batch_count)
+
+        idf_dict: Dict[int, float] = defaultdict(
+            lambda: log((num_docs + 1) / (1))
+        )
+        idf_dict.update(
+            {
+                idx: log((num_docs + 1) / (count + 1))
+                for idx, count in idf_count.items()
+            }
+        )
+        return idf_dict, collected_input_ids
+
+def ModernBERTBaseScore(**kwargs: Any) -> "BertScore":
+    """BertScore with ModernBERT-base-ModBERTScore-12"""
+    kwargs.pop("model_id", None)
+    return BertScore(model_id="LazerLambda/ModernBERT-base-ModBERTScore-12", **kwargs)
+
+
+def ModernBERTLargeScore(**kwargs: Any) -> "BertScore":
+    """BertScore with ModernBERT-large-ModBERTScore-19"""
+    kwargs.pop("model_id", None)
+    return BertScore(model_id="LazerLambda/ModernBERT-large-ModBERTScore-19", **kwargs)
+
+
+def RobertaBaseScore(**kwargs: Any) -> "BertScore":
+    """BertScore with roberta-base-ModBERTScore-10"""
+    kwargs.pop("model_id", None)
+    return BertScore(model_id="LazerLambda/roberta-base-ModBERTScore-10", **kwargs)
+
+
+def RobertaLargeScore(**kwargs: Any) -> "BertScore":
+    """BertScore with roberta-large-ModBERTScore-17"""
+    kwargs.pop("model_id", None)
+    return BertScore(model_id="LazerLambda/roberta-large-ModBERTScore-17", **kwargs)
+
+
+def RobertaLargeMNLIScore(**kwargs: Any) -> "BertScore":
+    """BertScore with roberta-large-mnli-ModBERTScore-19"""
+    kwargs.pop("model_id", None)
+    return BertScore(model_id="LazerLambda/roberta-large-mnli-ModBERTScore-19", **kwargs)

modern_bert_score-0.0.1/modern_bert_score/consts.py

@@ -0,0 +1,8 @@
+
+BASELINES = {
+    "LazerLambda/ModernBERT-base-ModBERTScore-12": (0.7899982,0.790043,0.7898656),
+    "LazerLambda/ModernBERT-large-ModBERTScore-19": (0.64403415,0.64410657,0.64355606),
+    "LazerLambda/roberta-base-ModBERTScore-10": (0.80465585,0.80464727,0.8043641),
+    "LazerLambda/roberta-large-ModBERTScore-17": (0.8243552,0.8243522,0.8240494),
+    "LazerLambda/roberta-large-mnli-ModBERTScore-19": (0.6901594,0.69018,0.6896288)
+}

modern_bert_score-0.0.1/modern_bert_score/inference.py

@@ -0,0 +1,139 @@
+import gc
+from typing import Any, List
+
+import torch
+from sentence_transformers import SentenceTransformer
+from torch.nn import functional as F
+from transformers import AutoTokenizer
+
+try:
+    from vllm import LLM
+
+    VLLM_AVAILABLE = True
+except ImportError:
+    LLM = object  # To prevent NameError if vllm is not installed
+    VLLM_AVAILABLE = False
+
+
+# TODO: Cache reference embeddings
+class Inference:
+
+    model: Any = None
+
+    def inference(
+        self, candidates: List[str], references: List[str], **kwargs: Any
+    ) -> tuple[List[torch.Tensor], List[torch.Tensor]]:
+        raise NotImplementedError("Method must be implemented in Subclass.")
+
+
+class STInference(Inference):
+    def __init__(
+        self,
+        model_id: str,
+        device: str = "cpu",
+        batch_size: int = 64,
+        **kwargs: Any,
+    ):
+        self.model = SentenceTransformer(
+            model_name_or_path=model_id, device=device, **kwargs
+        )
+        self.tokenizer = AutoTokenizer.from_pretrained(
+            model_id
+        )  # TODO: Maybe switch to PreTrainedTokenizerFast for clarity?
+        self.batch_size = batch_size
+        self.eps: float = 1e-12
+
+    def inference(
+        self, candidates: List[str], references: List[str], **kwargs: Any
+    ) -> tuple[List[torch.Tensor], List[torch.Tensor]]:
+
+        if self.model is None:
+            raise RuntimeError("Model not loaded.")
+        embds_refs = self.model.encode(
+            references,
+            output_value="token_embeddings",
+            convert_to_tensor=True,
+            **kwargs,
+        )
+        embds_refs = [
+            F.normalize(e, p=2, dim=-1, eps=self.eps) for e in embds_refs
+        ]
+        embds_cnds = self.model.encode(
+            candidates,
+            output_value="token_embeddings",
+            convert_to_tensor=True,
+            **kwargs,
+        )
+        embds_cnds = [
+            F.normalize(e, p=2, dim=-1, eps=self.eps) for e in embds_cnds
+        ]
+
+        return embds_cnds, embds_refs
+
+
+class VLLMInference(Inference):
+    def __init__(self, **kwargs: Any):
+        if not VLLM_AVAILABLE:
+            raise ImportError(
+                "vLLM is not installed. To use the vLLM backend, please "
+                "install it with `pip install vllm` or "
+                "`pip install 'modern-bert-score[vllm]'`."
+            )
+        # Backward compatibility for old callsites that pass task="embed".
+        kwargs = self._prepare_args(kwargs)
+        try:
+            self.model = LLM(**kwargs)
+        except Exception as exc:
+            message = str(exc)
+            if (
+                "Model architectures" in message
+                and "ModernBertForMaskedLM" in message
+            ):
+                raise RuntimeError(
+                    "vLLM does not accept the masked-LM ModernBERT checkpoint "
+                    "directly. Export an encoder-only checkpoint first with "
+                    "prepare_model.py, which rewrites the saved config to "
+                    "advertise ModernBertModel, then load that local path in "
+                    "VLLMInference. If you do not need vLLM specifically, use "
+                    "STInference for the original HF checkpoint."
+                ) from exc
+            raise
+        self.eps: float = 1e-12
+
+    @staticmethod
+    def _prepare_args(kwargs: Any) -> Any:
+        task = kwargs.pop("task", None)
+        if task == "embed":
+            kwargs.setdefault("runner", "pooling")
+            kwargs.setdefault("convert", "embed")
+        return kwargs
+
+    def inference(
+        self, candidates: List[str], references: List[str], **kwargs: Any
+    ) -> tuple[List[torch.Tensor], List[torch.Tensor]]:
+        if self.model is None:
+            raise RuntimeError("Model not loaded.")
+        outputs_cands = self.model.encode(
+            candidates, pooling_task="token_embed", **kwargs
+        )
+        outputs_refs = self.model.encode(
+            references, pooling_task="token_embed", **kwargs
+        )
+        collector: List[torch.Tensor] = []
+        for output in outputs_cands:
+            embeds = output.outputs.data
+            collector.append(embeds)
+        for output in outputs_refs:
+            embeds = output.outputs.data
+            collector.append(embeds)
+
+        collector = [
+            F.normalize(e, p=2, dim=-1, eps=self.eps) for e in collector
+        ]  # TODO: Check superflous?
+        return collector[0 : len(candidates)], collector[len(candidates) :]
+
+    def cleanup(self) -> None:
+        if hasattr(self, "model") and self.model:
+            del self.model
+        gc.collect()
+        torch.cuda.empty_cache()

modern_bert_score-0.0.1/modern_bert_score.egg-info/PKG-INFO

@@ -0,0 +1,124 @@
+Metadata-Version: 2.4
+Name: modern-bert-score
+Version: 0.0.1
+Summary: A reimplementation of the BERTScore metric optimized for modern inference workflows.
+Author-email: Philipp Koch <PhillKoch@protonmail.com>
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/pypa/sampleproject
+Project-URL: Issues, https://github.com/pypa/sampleproject/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Environment :: GPU :: NVIDIA CUDA
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE.md
+Requires-Dist: sentence-transformers>=5.2.3
+Requires-Dist: torch>=2.9.1
+Requires-Dist: transformers>=4.57.6
+Provides-Extra: vllm
+Requires-Dist: vllm>=0.16.0; extra == "vllm"
+Provides-Extra: dev
+Requires-Dist: sphinx; extra == "dev"
+Requires-Dist: sphinx_rtd_theme; extra == "dev"
+Requires-Dist: myst-parser; extra == "dev"
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: flake8; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+Requires-Dist: flake8; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Dynamic: license-file
+
+# Modern BERTScore for Fast Inference
+
+[![CI](https://github.com/LazerLambda/modern-bert-score/actions/workflows/ci.yml/badge.svg)](https://github.com/LazerLambda/modern-bert-score/actions/workflows/ci.yml)
+[![Python 3.12](https://img.shields.io/badge/python-3.12-blue.svg)](https://www.python.org/downloads/release/python-3120/)
+[![Python 3.13](https://img.shields.io/badge/python-3.13-blue.svg)](https://www.python.org/downloads/release/python-3130/)
+[![Python 3.14](https://img.shields.io/badge/python-3.14-blue.svg)](https://www.python.org/downloads/)
+[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+
+![Performance](runtime_comp_nvidia_rtx_4060.png)
+
+**Modern-BERT-Score** is a reimplementation of the BERTScore metric introduced by [Zhang et al., 2019](https://arxiv.org/abs/1904.09675), optimized for modern inference workflows using [SentenceTransformers](https://www.sbert.net/) and [vLLM](https://vllm.ai/).  
+
+This library provides fast, GPU-accelerated scoring for text generation evaluation, making BERTScore practical for large-scale inference tasks.
+
+---
+
+## ⚡ Features
+- Fast, efficient computation with optional vLLM support  
+- Compatible with all Hugging Face transformer models  
+- Supports truncated and optimized model versions for faster inference  
+- Works seamlessly with both CPU and GPU setups  
+
+---
+
+## 📦 Installation
+
+Modern-BERT-Score comes in **two variants**: a base version and a vLLM-enhanced version. For vLLM, an NVIDIA GPU is strongly recommended.  
+
+### Base Version
+```bash
+pip install modern-bert-score
+```
+
+### vLLM Version
+```{bash}
+pip install modern-bert-score[vllm]
+```
+This implementation is significantly faster than the original BERTScore, especially with GPU acceleration.
+
+## 📝 BERTScore
+BERTScore ([Zhang et al., 2019](https://arxiv.org/abs/1904.09675)) evaluates the similarity between candidate and reference texts by comparing their contextual embeddings from a pre-trained transformer model. For each token in the candidate, it finds the most similar token in the reference (using cosine similarity) and aggregates these scores to compute **precision**, **recall**, and **F1**. Optionally, **IDF-weighting** can be applied to give more importance to rare and informative words, improving the metric’s sensitivity to meaningful content over common words. Additionally, optional **Baseline Rescaling** shifts the scores such that the score is in the range of [0,1]. This approach captures semantic similarity beyond exact word matches, making it robust for tasks such as machine translation and text generation evaluation.  
+
+The following figure, taken from the original paper, illustrates how BERTScore works:
+
+![BERTScore](zhang_19_figure_1.png)
+
+## 🛠 Usage
+### Example
+```python
+from modern_bert_score import BertScore
+
+candidates = ["Hello World!", "A robin is a bird."]
+references = ["Hi World!", "A robin is not a bird."]
+
+metric = BertScore(model_id="roberta-base")
+scores = metric(candidates, references)
+
+# scores is a list of (Precision, Recall, F1) tuples
+# To get separate lists of P, R, F1:
+P, R, F1 = zip(*scores)
+
+print("Precision scores:", P)
+print("Recall scores:", R)
+print("F1 scores:", F1)
+```
+
+## ⚠️ NOTICE
+
+- For best performance, an optimal layer should be used for each model.  
+- To find the optimal layer, [please use this script from the original BERTScore implementation](https://github.com/Tiiiger/bert_score/tree/master/tune_layers).  
+
+Some pre-truncated models optimized for vLLM are available on [Hugging Face](https://huggingface.co/collections/LazerLambda/modern-bertscore) and directly available in this library:
+
+- `LazerLambda/ModernBERT-base-ModBERTScore-12` -> `ModernBERTBaseScore`
+- `LazerLambda/ModernBERT-large-ModBERTScore-19` -> `ModernBERTLargeScore` 
+- `LazerLambda/roberta-base-ModBERTScore-10`  -> `RobertaBaseScore`
+- `LazerLambda/roberta-large-ModBERTScore-17` -> `RobertaLargeScore`
+- `LazerLambda/roberta-large-mnli-ModBERTScore-19` -> `RobertaLargeMNLIScore`
+
+
+## 🗺 Roadmap
+
+- [x] Implement base version and vLLM addon  
+- [x] Add IDF-weighted scoring  
+- [ ] Add baseline-rescaling and scripts for identifying optimal baselines  
+- [ ] Add model (vLLM-)adaptation script for slicing the model
+- [ ] Add multilingual support
+- [ ] Add CLI tool

modern_bert_score-0.0.1/modern_bert_score.egg-info/SOURCES.txt

@@ -0,0 +1,13 @@
+LICENSE.md
+README.md
+pyproject.toml
+modern_bert_score/__init__.py
+modern_bert_score/bert_score.py
+modern_bert_score/consts.py
+modern_bert_score/inference.py
+modern_bert_score.egg-info/PKG-INFO
+modern_bert_score.egg-info/SOURCES.txt
+modern_bert_score.egg-info/dependency_links.txt
+modern_bert_score.egg-info/requires.txt
+modern_bert_score.egg-info/top_level.txt
+tests/test_bert_score.py

modern_bert_score-0.0.1/modern_bert_score.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

modern_bert_score-0.0.1/modern_bert_score.egg-info/requires.txt

@@ -0,0 +1,16 @@
+sentence-transformers>=5.2.3
+torch>=2.9.1
+transformers>=4.57.6
+
+[dev]
+sphinx
+sphinx_rtd_theme
+myst-parser
+pytest
+pytest-cov
+ruff
+flake8
+mypy
+
+[vllm]
+vllm>=0.16.0

modern_bert_score-0.0.1/modern_bert_score.egg-info/top_level.txt

@@ -0,0 +1 @@
+modern_bert_score

modern_bert_score-0.0.1/pyproject.toml

@@ -0,0 +1,66 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "modern-bert-score"
+version = "0.0.1"
+description = "A reimplementation of the BERTScore metric optimized for modern inference workflows."
+readme = "README.md"
+requires-python = ">=3.12"
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Operating System :: OS Independent",
+    "Development Status :: 2 - Pre-Alpha",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "Environment :: GPU :: NVIDIA CUDA",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence"
+]
+license = "Apache-2.0"
+license-files = ["LICEN[CS]E*"]
+authors = [
+    {name = "Philipp Koch", email="PhillKoch@protonmail.com"},
+]
+dependencies = [
+    "sentence-transformers>=5.2.3",
+    "torch>=2.9.1",
+    "transformers>=4.57.6",
+]
+
+[project.urls]
+Homepage = "https://github.com/pypa/sampleproject"
+Issues = "https://github.com/pypa/sampleproject/issues"
+
+[project.optional-dependencies]
+vllm = ["vllm>=0.16.0"]
+dev = [
+    "sphinx",
+    "sphinx_rtd_theme",
+    "myst-parser",
+    "pytest",
+    "pytest-cov",
+    "ruff",
+    "flake8",
+    "mypy",
+    "flake8",
+    "pytest-cov"
+]
+
+[tool.setuptools.packages.find]
+include = ["modern_bert_score*"]
+
+[tool.ruff]
+line-length = 90
+extend-exclude = ["tests"]
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I"]
+
+[tool.mypy]
+strict = true
+ignore_missing_imports = true
+
+[tool.pytest.ini_options]
+filterwarnings = ["ignore::DeprecationWarning"]
+exclude = ["^build/"]

modern_bert_score-0.0.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

modern_bert_score-0.0.1/tests/test_bert_score.py

@@ -0,0 +1,326 @@
+import unittest
+from unittest.mock import MagicMock, patch
+
+import pytest
+import torch
+from transformers import logging as transformers_logging
+
+from modern_bert_score.bert_score import BertScore
+from modern_bert_score.inference import VLLM_AVAILABLE
+
+# Suppress transformers warnings about missing weights during testing
+transformers_logging.set_verbosity_error()
+
+TEST_MODEL= "LazerLambda/BERT-Tiny-L-2-H-128-A-2-ModBERTScore-TEST"
+
+class TestBertScore(unittest.TestCase):
+
+    test_model: str = TEST_MODEL
+
+    def setUp(self):
+        self.candidates = [
+            "Hello, my name is",
+            "The president of the United States is",
+            "The capital of France is",
+            "The future of AI is",
+            "The cat is on the table",
+        ]
+        self.references = [
+            "Hello, my name is",
+            "The head of the United States is",
+            "The capital of Japan is",
+            "The future of Work is",
+            "The dog is on the table",
+        ]
+
+    def test_against_original(self):
+        """Test basic functionality."""
+        original_p_r_f1 = [
+            torch.tensor([1.0000, 0.9423, 0.8705, 0.9105, 0.9637]),
+            torch.tensor([1.0000, 0.9423, 0.8705, 0.9105, 0.9637]),
+            torch.tensor([1.0000, 0.9423, 0.8705, 0.9105, 0.9637])
+        ]
+        original_p_r_f1 = torch.stack(original_p_r_f1).T
+        original_p_r_f1_idf = [
+            torch.tensor([1.0000, 0.9233, 0.8167, 0.8609, 0.9369]),
+            torch.tensor([1.0000, 0.9358, 0.8304, 0.8861, 0.9505]),
+            torch.tensor([1.0000, 0.9295, 0.8235, 0.8733, 0.9436])
+        ]
+        original_p_r_f1_idf = torch.stack(original_p_r_f1_idf).T
+        bs = BertScore(model_id=self.test_model, backend="default")
+        p_r_f1 = bs(self.candidates, self.references)
+        bs.idf_weighting = True
+        p_r_f1_idf = bs(self.candidates, self.references)
+        for (p, r, f1), (p_exp, r_exp, f1_exp) in zip(p_r_f1, original_p_r_f1):
+            self.assertTrue(torch.allclose(torch.tensor(p), p_exp, atol=1e-4))
+            self.assertTrue(torch.allclose(torch.tensor(r), r_exp, atol=1e-4))
+            self.assertTrue(torch.allclose(torch.tensor(f1), f1_exp, atol=1e-4))
+        for (p, r, f1), (p_exp, r_exp, f1_exp) in zip(p_r_f1_idf, original_p_r_f1_idf):
+            self.assertTrue(torch.allclose(torch.tensor(p), p_exp, atol=1e-4))
+            self.assertTrue(torch.allclose(torch.tensor(r), r_exp, atol=1e-4))
+            self.assertTrue(torch.allclose(torch.tensor(f1), f1_exp, atol=1e-4))
+
+    def test_id(self):
+        cand1 = ["Hello World!"]
+        ref1 = ["Hello World!"]
+        bs = BertScore(model_id=self.test_model, backend="default")
+        p_r_f1 = bs(cand1, ref1)
+        self.assertAlmostEqual(p_r_f1[0][0], 1.0, places=5)
+        self.assertAlmostEqual(p_r_f1[0][1], 1.0, places=5)
+        self.assertAlmostEqual(p_r_f1[0][2], 1.0, places=5)
+
+        cand2 = ["Hello World!"]
+        ref2 = [" Hello World! "]
+        p_r_f1 = bs(cand2, ref2)
+        self.assertAlmostEqual(p_r_f1[0][0], 1.0, places=5)
+        self.assertAlmostEqual(p_r_f1[0][1], 1.0, places=5)
+        self.assertAlmostEqual(p_r_f1[0][2], 1.0, places=5)
+
+    def test_unequal_length(self):
+        cand1 = ["Hello World!"]
+        ref1 = ["Hello World!", "Hello World!"]
+        bs = BertScore(model_id=self.test_model, backend="default")
+        with self.assertRaises(ValueError):
+            bs(cand1, ref1)
+
+        cand2 = ["Hello World!", "Hello World!"]
+        ref2 = ["Hello World!"]
+        bs = BertScore(model_id=self.test_model, backend="default")
+        with self.assertRaises(ValueError):
+            bs(cand2, ref2)
+
+    def test_unequal_input(self):
+        cand1 = ["Hello World!"]
+        ref1 = ["Bye World!"]
+        bs = BertScore(model_id=self.test_model, backend="default")
+        p_r_f1 = bs(cand1, ref1)
+        self.assertTrue(p_r_f1[0][0] < 1.0)
+        self.assertTrue(p_r_f1[0][1] < 1.0)
+        self.assertTrue(p_r_f1[0][2] < 1.0)
+
+    def test_empty_input(self):
+        cand1 = []
+        ref1 = []
+        bs = BertScore(model_id=self.test_model, backend="default")
+        p_r_f1 = bs(cand1, ref1)
+        self.assertEqual(p_r_f1, [])
+
+    def test_empty_inference_engine(self):
+        bs = BertScore(model_id=self.test_model, backend="default")
+        bs.inference_engine = None
+        with self.assertRaises(ValueError):
+             bs(self.candidates, self.references)
+
+    def test_check_nan(self):
+        f1_nan = torch.tensor(torch.nan)
+        f1_checked = BertScore._check_nan(f1_nan)
+        self.assertEqual(f1_checked, torch.Tensor([0.0]))
+
+    def test_exception_idf_bertscore(self):
+        bs = BertScore(model_id=self.test_model, backend="default")
+        with self.assertRaises(ValueError):
+            bs.bert_score(
+                candidates=torch.rand(3, 4),
+                references=torch.rand(3, 4),
+                input_ids_cand=[0, 1, 2],
+                input_ids_ref=[0, 1, 2],
+            )
+
+    def test_single(self):
+        cand1 = "Hello World!"
+        ref1 = "Hello World!"
+        bs = BertScore(model_id=self.test_model, backend="default")
+        bs(cand1, ref1)
+
+    def test_tokenize_batch(self):
+        bs = BertScore(model_id=self.test_model, backend="default")
+        counter, input_ids = bs._process_batch(
+            ["Hello World!", "Hello World!"],
+            bs.tokenizer,
+            ignore_counter=True
+        )
+        assert len(counter) == 0
+        counter, input_ids = bs._process_batch(
+            ["Hello World!", "Hello World!"],
+            bs.tokenizer,
+            ignore_counter=False
+        )
+        assert counter is not None
+
+    def test_tokenize_data(self):
+        bs = BertScore(model_id=self.test_model, backend="default")
+        bs._tokenize_data(["Hello World!", "Hello World!"], nthreads=4)
+        bs._tokenize_data(["Hello World!", "Hello World!"], nthreads=0)
+
+    def test_get_idf_dict(self):
+        bs = BertScore(model_id=self.test_model, backend="default")
+        bs.get_idf_dict(["Hello World!", "Hello World!"], nthreads=4)
+        bs.get_idf_dict(["Hello World!", "Hello World!"], nthreads=0)
+
+    def test_base_line(self):
+        cand1 = ["Hello World!"]
+        ref1 = ["Hello World!"]
+        bs = BertScore(model_id=self.test_model, backend="default", baseline_rescaling=True, custom_baseline=(0.5, 0.5, 0.5))
+        p_r_f1 = bs(cand1, ref1)
+        self.assertAlmostEqual(p_r_f1[0][0], 1.0, places=5)
+        self.assertAlmostEqual(p_r_f1[0][1], 1.0, places=5)
+        self.assertAlmostEqual(p_r_f1[0][2], 1.0, places=5)
+
+    def test_baseline_rescaling_exception(self):
+        """Test that ValueError is raised when baseline is missing."""
+        with self.assertRaises(ValueError):
+            BertScore(model_id=self.test_model, backend="default", baseline_rescaling=True)
+
+    def test_baseline_rescaling_injection(self):
+        """Test baseline rescaling with injected baseline for the test model."""
+        from modern_bert_score.consts import BASELINES
+        # Inject test model baseline (P, R, F1) tuple. F1 is at index 2.
+        BASELINES[self.test_model] = (0.5, 0.5, 0.5)
+        try:
+            bs = BertScore(model_id=self.test_model, backend="default", baseline_rescaling=True)
+            self.assertEqual(bs.baseline, (0.5, 0.5, 0.5))
+            # Test computation
+            cand1 = ["Hello World!"]
+            ref1 = ["Hello World!"]
+            p_r_f1 = bs(cand1, ref1)
+            self.assertAlmostEqual(p_r_f1[0][2], 1.0, places=5)
+        finally:
+            del BASELINES[self.test_model]
+
+    def test_inference_base_not_implemented_error(self):
+        from modern_bert_score.inference import Inference
+        with self.assertRaises(NotImplementedError):
+            dummy = Inference()
+            dummy.inference(["Hello"], ["Hello"])
+
+    @patch("modern_bert_score.bert_score.AutoTokenizer")
+    @patch("modern_bert_score.bert_score.STInference")
+    def test_initialization_default(self, mock_st_inference, mock_tokenizer):
+        """Test initialization with default backend."""
+        # Setup mocks
+        mock_tokenizer.from_pretrained.return_value = MagicMock()
+
+        bs = BertScore(model_id=self.test_model, backend="default")
+
+        mock_tokenizer.from_pretrained.assert_called_with(self.test_model)
+        mock_st_inference.assert_called_once()
+        self.assertEqual(bs.inference_engine, mock_st_inference.return_value)
+
+    @patch("modern_bert_score.bert_score.AutoTokenizer")
+    @patch("modern_bert_score.bert_score.VLLMInference")
+    def test_initialization_vllm(self, mock_vllm_inference, mock_tokenizer):
+        """Test initialization with vllm backend."""
+        # Setup mocks
+        mock_tokenizer.from_pretrained.return_value = MagicMock()
+
+        bs = BertScore(model_id=self.test_model, backend="vllm")
+
+        mock_vllm_inference.assert_called_once()
+        self.assertEqual(bs.inference_engine, mock_vllm_inference.return_value)
+
+    @patch("modern_bert_score.bert_score.AutoTokenizer")
+    @patch("modern_bert_score.inference.VLLM_AVAILABLE", False)
+    def test_initialization_vllm_without_vllm_installed(self, mock_tokenizer):
+        """Test that using vllm backend without vllm installed raises ImportError."""
+        mock_tokenizer.from_pretrained.return_value = MagicMock()
+        with self.assertRaises(ImportError) as cm:
+            BertScore(model_id=self.test_model, backend="vllm")
+
+        self.assertIn("vLLM is not installed", str(cm.exception))
+        self.assertIn("pip install 'modern-bert-score[vllm]'", str(cm.exception))
+
+    @patch("modern_bert_score.inference.LLM")
+    @patch("modern_bert_score.inference.VLLM_AVAILABLE", True)
+    @patch("modern_bert_score.bert_score.AutoTokenizer")
+    def test_initialization_vllm_masked_lm_error(self, mock_tokenizer, mock_llm):
+        """Test that appropriate error is raised when vLLM rejects MaskedLM architecture.
+
+        This test uses mocks for vLLM components, so it will run (and pass) regardless
+        of whether the `vllm` package is installed in the test environment.
+        """
+        mock_tokenizer.from_pretrained.return_value = MagicMock()
+
+        # Simulate vLLM raising exception about architecture
+        mock_llm.side_effect = Exception(
+            "ValueError: Model architectures ['ModernBertForMaskedLM'] are not supported for now. "
+            "Supported architectures: ['ModernBertModel', ...]"
+        )
+
+        with self.assertRaises(RuntimeError) as cm:
+            BertScore(model_id=self.test_model, backend="vllm")
+
+        self.assertIn(
+            "vLLM does not accept the masked-LM ModernBERT checkpoint directly",
+            str(cm.exception),
+        )
+
+    @patch("modern_bert_score.bert_score.AutoTokenizer")
+    def test_initialization_invalid_backend(self, mock_tokenizer):
+        """Test initialization with invalid backend raises ValueError."""
+        with self.assertRaises(ValueError):
+            BertScore(model_id=self.test_model, backend="invalid")
+
+    @patch("modern_bert_score.bert_score.AutoTokenizer")
+    @patch("modern_bert_score.bert_score.STInference")
+    def test_call_simple(self, mock_st_inference, mock_tokenizer):
+        """Test scoring call with default backend."""
+        mock_engine = MagicMock()
+        mock_st_inference.return_value = mock_engine
+
+        # Mock embeddings: List[Tensor]
+        # Need shape [seq_len, hidden_dim].
+        # BertScore slices [1:-1] (removes CLS and SEP), so we need at least 3 tokens.
+        c_emb = torch.rand(3, 4)
+        r_emb = torch.rand(3, 4)
+
+        mock_engine.inference.return_value = ([c_emb], [r_emb])
+
+        bs = BertScore(model_id=self.test_model, backend="default")
+        results = bs(self.candidates, self.references)
+
+        self.assertEqual(len(results), 1)
+        p, r, f1 = results[0]
+        self.assertIsInstance(p, float)
+        self.assertIsInstance(r, float)
+        self.assertIsInstance(f1, float)
+
+        # Verify inference called correctly
+        mock_engine.inference.assert_called_with(self.candidates, self.references)
+
+    @patch("modern_bert_score.bert_score.AutoTokenizer")
+    @patch("modern_bert_score.bert_score.STInference")
+    def test_input_validation(self, mock_st_inference, mock_tokenizer):
+        """Test input validation for candidates/references."""
+        bs = BertScore(model_id=self.test_model, backend="default")
+
+        # Mismatched lengths
+        with self.assertRaises(ValueError):
+            bs(["a"], ["b", "c"])
+
+@pytest.fixture(scope="module")
+def vllm_bert_score():
+    # This setup runs once for the module
+    bs = BertScore(
+        model_id=TEST_MODEL,
+        backend="vllm",
+        vllm_args={"gpu_memory_utilization": 0.3, "enforce_eager": True, "distributed_executor_backend": "mp", "task": "embed"}
+    )
+    return bs
+
+# 2. The actual test function
+@pytest.mark.skipif(not VLLM_AVAILABLE, reason="vLLM not installed")
+def test_vllm_only_feature(vllm_bert_score):
+    """Test that requests the fixture above."""
+    cand1 = ["Hello World!"]
+    ref1 = ["Hello World!"]
+
+    # Use the fixture passed as an argument
+    p_r_f1 = vllm_bert_score(cand1, ref1)
+
+    # Use standard asserts instead of self.assertEqual
+    assert p_r_f1[0][0] == pytest.approx(1.0)
+    assert p_r_f1[0][1] == pytest.approx(1.0)
+    assert p_r_f1[0][2] == pytest.approx(1.0)
+
+    # Test kwargs passed to vLLMInference
+    # kwargs = {"task": "embed", "gpu_memory_utilization": 0.3, "enforce_eager": True, "distributed_executor_backend": "mp"}