aidial-rag-eval 0.6.0.dev16__tar.gz → 0.6.0.dev20__tar.gz

{aidial_rag_eval-0.6.0.dev16 → aidial_rag_eval-0.6.0.dev20}/PKG-INFO

@@ -1,13 +1,13 @@
 Metadata-Version: 2.4
 Name: aidial-rag-eval
-Version: 0.6.0.dev16
+Version: 0.6.0.dev20
 Summary: Library for RAG evaluation
 License: Apache-2.0
 License-File: LICENSE
 Keywords: ai
 Author: EPAM RAIL
 Author-email: SpecialEPM-DIALDevTeam@epam.com
-Requires-Python: >=3.11,<4.0
+Requires-Python: >=3.11,<3.15
 Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.11
@@ -19,12 +19,12 @@ Provides-Extra: s3fs
 Requires-Dist: fsspec (>=2024.3.1,<2025.0.0)
 Requires-Dist: langchain-core (>=0.3.80,<2)
 Requires-Dist: more-itertools (>=10.5.0,<11.0.0)
-Requires-Dist: nltk (>=3.8.1,<4.0.0)
 Requires-Dist: numpy (>=1.26.4,<3)
 Requires-Dist: pandas (>=2.2.2,<3.0.0)
 Requires-Dist: pyarrow (>=20.0.0)
-Requires-Dist: pydantic (>=2.7.1,<3.0.0)
+Requires-Dist: pydantic (>=2.7.1)
 Requires-Dist: s3fs (>=2024.3.1,<2025.0.0) ; extra == "s3fs"
+Requires-Dist: spacy (>=3.8.9,<4)
 Requires-Dist: tqdm (>=4.66.4,<5.0.0)
 Project-URL: Documentation, https://github.com/epam/ai-dial-rag-eval
 Project-URL: Homepage, https://github.com/epam/ai-dial-rag-eval
@@ -47,6 +47,27 @@ Install the library using [pip](https://pip.pypa.org/en/stable/getting-started):
 pip install aidial-rag-eval
 ```
 
+### spaCy language model
+
+The generation metrics require the English language model for [spaCy](https://spacy.io/).
+Download it after installation:
+
+```sh
+python -m spacy download en_core_web_sm
+```
+
+Alternatively, you can install the model directly via URL:
+
+```sh
+pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl
+```
+
+Or as a [Poetry](https://python-poetry.org/) dependency:
+
+```toml
+en-core-web-sm = {url = "https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl"}
+```
+
 ### Example
 
 The example of how to get retrieval metrics along with answer inference based on the context.
@@ -56,7 +77,8 @@ import pandas as pd
 from langchain_openai import AzureChatOpenAI
 from aidial_rag_eval import create_rag_eval_metrics_report
 from aidial_rag_eval.metric_binds import CONTEXT_TO_ANSWER_INFERENCE,\
-    ANSWER_TO_GROUND_TRUTH_INFERENCE, GROUND_TRUTH_TO_ANSWER_INFERENCE
+    ANSWER_TO_GROUND_TRUTH_INFERENCE, GROUND_TRUTH_TO_ANSWER_INFERENCE,\
+    ANSWER_TO_FACTS_INFERENCE, FACTS_TO_ANSWER_INFERENCE
 
 llm = AzureChatOpenAI(model="gemini-2.5-flash-lite")
 
@@ -87,16 +109,18 @@ df_metrics = create_rag_eval_metrics_report(
         CONTEXT_TO_ANSWER_INFERENCE,
         ANSWER_TO_GROUND_TRUTH_INFERENCE,
         GROUND_TRUTH_TO_ANSWER_INFERENCE,
+        ANSWER_TO_FACTS_INFERENCE,
+        FACTS_TO_ANSWER_INFERENCE,
     ],
 )
-print(df_metrics[["facts_ranks", "recall", 'precision', 'mrr', 'f1', 'ctx_ans_inference', 'ans_gt_inference', 'gt_ans_inference']])
+print(df_metrics[["facts_ranks", "recall", 'precision', 'mrr', 'f1', 'ctx_ans_inference', 'ans_gt_inference', 'gt_ans_inference', 'ans_fct_inference', 'fct_ans_inference']])
 ```
 
 It is expected to see the following results:
 
-| recall | precision | mrr | f1  | ctx_ans_inference | ans_gt_inference | gt_ans_inference |
-| ------ | --------- | --- | --- | ----------------- | ---------------- | ---------------- |
-| 0.5    | 0.5       | 0.5 | 0.5 | 1.0               | 0.5              | 1.0              |
+| recall | precision | mrr | f1  | ctx_ans_inference | ans_gt_inference | gt_ans_inference | ans_fct_inference | fct_ans_inference |
+| ------ | --------- | --- | --- | ----------------- | ---------------- | ---------------- | ----------------- | ----------------- |
+| 0.5    | 0.5       | 0.5 | 0.5 | 1.0               | 0.5              | 1.0              | 0.5               | 1.0               |
 
 In this table:
 
@@ -106,17 +130,22 @@ In this table:
   - "ctx" refers to 'context'
   - "ans" refers to 'answer'
   - "gt" refers to 'ground truth answer'
-- "ctx_ans_inference" and "ans_gt_inference" values of 1.0 mean our answer can be derived directly from the context and the ground truth answer, respectively.
-- "gt_ans_inference" of 0.5, denotes that the ground truth answer can only be partially inferred from our answer.
+  - "fct" refers to 'facts'
+- "ctx_ans_inference" of 1.0 means our answer can be fully derived from the context.
+- "ans_gt_inference" of 0.5 means the ground truth answer is only partially entailed by our answer.
+- "gt_ans_inference" of 1.0 means our answer can be fully derived from the ground truth answer.
+- "ans_fct_inference" of 0.5 means only half of the ground truth facts are entailed by our answer (the Pacific Ocean fact is missing).
+- "fct_ans_inference" of 1.0 means our answer can be fully derived from the ground truth facts.
 
 ## Recommended models
 
 The algorithm is token-intensive. Considering the balance between quality and price, the following models are recommended:
 
+- **gemini-3.1-flash-lite**
 - **gemini-2.5-flash-lite**
 - **gpt-5-mini**
-- **gemini-2.0-flash-lite**
 - **gpt-5-nano**
+- **gpt-5.4-mini**
 
 ## Developer environment
 

{aidial_rag_eval-0.6.0.dev16 → aidial_rag_eval-0.6.0.dev20}/README.md

@@ -14,6 +14,27 @@ Install the library using [pip](https://pip.pypa.org/en/stable/getting-started):
 pip install aidial-rag-eval
 ```
 
+### spaCy language model
+
+The generation metrics require the English language model for [spaCy](https://spacy.io/).
+Download it after installation:
+
+```sh
+python -m spacy download en_core_web_sm
+```
+
+Alternatively, you can install the model directly via URL:
+
+```sh
+pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl
+```
+
+Or as a [Poetry](https://python-poetry.org/) dependency:
+
+```toml
+en-core-web-sm = {url = "https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl"}
+```
+
 ### Example
 
 The example of how to get retrieval metrics along with answer inference based on the context.
@@ -23,7 +44,8 @@ import pandas as pd
 from langchain_openai import AzureChatOpenAI
 from aidial_rag_eval import create_rag_eval_metrics_report
 from aidial_rag_eval.metric_binds import CONTEXT_TO_ANSWER_INFERENCE,\
-    ANSWER_TO_GROUND_TRUTH_INFERENCE, GROUND_TRUTH_TO_ANSWER_INFERENCE
+    ANSWER_TO_GROUND_TRUTH_INFERENCE, GROUND_TRUTH_TO_ANSWER_INFERENCE,\
+    ANSWER_TO_FACTS_INFERENCE, FACTS_TO_ANSWER_INFERENCE
 
 llm = AzureChatOpenAI(model="gemini-2.5-flash-lite")
 
@@ -54,16 +76,18 @@ df_metrics = create_rag_eval_metrics_report(
         CONTEXT_TO_ANSWER_INFERENCE,
         ANSWER_TO_GROUND_TRUTH_INFERENCE,
         GROUND_TRUTH_TO_ANSWER_INFERENCE,
+        ANSWER_TO_FACTS_INFERENCE,
+        FACTS_TO_ANSWER_INFERENCE,
     ],
 )
-print(df_metrics[["facts_ranks", "recall", 'precision', 'mrr', 'f1', 'ctx_ans_inference', 'ans_gt_inference', 'gt_ans_inference']])
+print(df_metrics[["facts_ranks", "recall", 'precision', 'mrr', 'f1', 'ctx_ans_inference', 'ans_gt_inference', 'gt_ans_inference', 'ans_fct_inference', 'fct_ans_inference']])
 ```
 
 It is expected to see the following results:
 
-| recall | precision | mrr | f1  | ctx_ans_inference | ans_gt_inference | gt_ans_inference |
-| ------ | --------- | --- | --- | ----------------- | ---------------- | ---------------- |
-| 0.5    | 0.5       | 0.5 | 0.5 | 1.0               | 0.5              | 1.0              |
+| recall | precision | mrr | f1  | ctx_ans_inference | ans_gt_inference | gt_ans_inference | ans_fct_inference | fct_ans_inference |
+| ------ | --------- | --- | --- | ----------------- | ---------------- | ---------------- | ----------------- | ----------------- |
+| 0.5    | 0.5       | 0.5 | 0.5 | 1.0               | 0.5              | 1.0              | 0.5               | 1.0               |
 
 In this table:
 
@@ -73,17 +97,22 @@ In this table:
   - "ctx" refers to 'context'
   - "ans" refers to 'answer'
   - "gt" refers to 'ground truth answer'
-- "ctx_ans_inference" and "ans_gt_inference" values of 1.0 mean our answer can be derived directly from the context and the ground truth answer, respectively.
-- "gt_ans_inference" of 0.5, denotes that the ground truth answer can only be partially inferred from our answer.
+  - "fct" refers to 'facts'
+- "ctx_ans_inference" of 1.0 means our answer can be fully derived from the context.
+- "ans_gt_inference" of 0.5 means the ground truth answer is only partially entailed by our answer.
+- "gt_ans_inference" of 1.0 means our answer can be fully derived from the ground truth answer.
+- "ans_fct_inference" of 0.5 means only half of the ground truth facts are entailed by our answer (the Pacific Ocean fact is missing).
+- "fct_ans_inference" of 1.0 means our answer can be fully derived from the ground truth facts.
 
 ## Recommended models
 
 The algorithm is token-intensive. Considering the balance between quality and price, the following models are recommended:
 
+- **gemini-3.1-flash-lite**
 - **gemini-2.5-flash-lite**
 - **gpt-5-mini**
-- **gemini-2.0-flash-lite**
 - **gpt-5-nano**
+- **gpt-5.4-mini**
 
 ## Developer environment
 

{aidial_rag_eval-0.6.0.dev16 → aidial_rag_eval-0.6.0.dev20}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "aidial-rag-eval"
-version = "0.6.0.dev16"
+version = "0.6.0.dev20"
 description = "Library for RAG evaluation"
 authors = ["EPAM RAIL <SpecialEPM-DIALDevTeam@epam.com>"]
 homepage = "https://github.com/epam/ai-dial-rag-eval"
@@ -13,13 +13,13 @@ repository = "https://github.com/epam/ai-dial-rag-eval"
 
 
 [tool.poetry.dependencies]
-python = "^3.11"
+python = ">=3.11, <3.15"
 numpy = ">=1.26.4,<3"
 pandas = "^2.2.2"
 fsspec = "^2024.3.1"
 s3fs = {version = "^2024.3.1", optional = true}
-pydantic = "^2.7.1"
-nltk = "^3.8.1"
+pydantic = ">=2.7.1"
+spacy = ">=3.8.9,<4"
 tqdm = "^4.66.4"
 langchain-core = ">=0.3.80,<2"
 more-itertools = "^10.5.0"
@@ -35,6 +35,7 @@ pytest-cov = "^4.1.0"
 pytest-mock = "^3.14.0"
 langchain-community = "^0.3.14"
 langchain-openai = "^0.2.1"
+en-core-web-sm = {url = "https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl"}
 
 [tool.poetry.group.lint.dependencies]
 black = "^24.3.0"

{aidial_rag_eval-0.6.0.dev16 → aidial_rag_eval-0.6.0.dev20}/src/aidial_rag_eval/dataframe/metrics.py

@@ -8,13 +8,20 @@ from aidial_rag_eval.dataframe.match_facts import (
     apply_matcher_to_merged_dataframe,
 )
 from aidial_rag_eval.dataframe.merge import merge_ground_truth_and_answers
-from aidial_rag_eval.generation.metric_binds import metric_binds_dict
-from aidial_rag_eval.generation.types import MetricBind, inference_column
+from aidial_rag_eval.generation.metric_binds import (
+    _wrapped_dataframe_inference,
+    _wrapped_dataframe_refusal,
+)
+from aidial_rag_eval.generation.types import (
+    InferenceMetricBind,
+    MetricBind,
+    RefusalMetricBind,
+    inference_column,
+)
 from aidial_rag_eval.retrieval.metrics import (
     calculate_metrics as calculate_metrics_by_row,
 )
 from aidial_rag_eval.retrieval.types import Matcher
-from aidial_rag_eval.types import MergedColumns
 
 
 def apply_metrics_to_matched_results(match_result_data: pd.DataFrame) -> pd.DataFrame:
@@ -114,7 +121,6 @@ def calculate_generation_metrics(
     metric_binds: List[MetricBind],
     max_concurrency: int = 8,
     show_progress_bar: bool = True,
-    auto_download_nltk: bool = True,
 ) -> pd.DataFrame:
     """
     Calculates RAG evaluation generation metrics from df_merged dataframe.
@@ -130,7 +136,7 @@ def calculate_generation_metrics(
         A Langchain chat model used for calculating generation metrics.
 
     metric_binds : List[MetricBind], optional, default=None
-        A list of string constants from `aidial_rag_eval.metric_binds`
+        A list of metric bind instances from `aidial_rag_eval.metric_binds`
         specifying the generation metrics.
 
     max_concurrency : int, default=8
@@ -145,20 +151,32 @@ def calculate_generation_metrics(
         Returns generation metrics dataframe.
     """
     df_merged_copy = df_merged.copy()
-    df_merged_copy[MergedColumns.JOINED_CONTEXT] = df_merged_copy[
-        MergedColumns.CONTEXT
-    ].apply(lambda x: "\n".join(x))
     metric_results = dict()
     for metric_bind in metric_binds:
-        metric_results.update(
-            metric_binds_dict[metric_bind](
+        if isinstance(metric_bind, InferenceMetricBind):
+            result = _wrapped_dataframe_inference(
+                df_merged=df_merged_copy,
+                premise_column=metric_bind.premise_column,
+                hypothesis_column=metric_bind.hypothesis_column,
+                llm=llm,
+                prefix=metric_bind.prefix,
+                premise_prefix_column=metric_bind.premise_prefix_column,
+                document_column=metric_bind.document_column,
+                max_concurrency=max_concurrency,
+                show_progress_bar=show_progress_bar,
+            )
+        elif isinstance(metric_bind, RefusalMetricBind):
+            result = _wrapped_dataframe_refusal(
                 df_merged=df_merged_copy,
+                answer_column=metric_bind.answer_column,
                 llm=llm,
+                prefix=metric_bind.prefix,
                 max_concurrency=max_concurrency,
                 show_progress_bar=show_progress_bar,
-                auto_download_nltk=auto_download_nltk,
-            ).to_dict(orient="series")
-        )
+            )
+        else:
+            raise ValueError(f"Unknown metric bind type: {type(metric_bind)}")
+        metric_results.update(result.to_dict(orient="series"))
     df_metrics = pd.DataFrame(data=metric_results)
     nli_columns = [
         column for column in metric_results.keys() if column.endswith(inference_column)
@@ -180,7 +198,6 @@ def create_generation_metrics_report(
     metric_binds: List[MetricBind],
     max_concurrency: int = 8,
     show_progress_bar: bool = True,
-    auto_download_nltk: bool = True,
 ) -> pd.DataFrame:
     """
     Calculates RAG evaluation generation metrics from input dataframes.
@@ -201,7 +218,7 @@ def create_generation_metrics_report(
         A Langchain chat model used for calculating generation metrics.
 
     metric_binds : List[MetricBind], optional, default=None
-        A list of string constants from `aidial_rag_eval.metric_binds`
+        A list of metric bind instances from `aidial_rag_eval.metric_binds`
         specifying the generation metrics.
 
     max_concurrency : int, default=8
@@ -223,7 +240,6 @@ def create_generation_metrics_report(
         metric_binds,
         max_concurrency,
         show_progress_bar,
-        auto_download_nltk,
     )
     return pd.merge(df_merged, df_metrics, left_index=True, right_index=True)
 
@@ -236,7 +252,6 @@ def create_rag_eval_metrics_report(
     metric_binds: Optional[List[MetricBind]] = None,
     max_concurrency: int = 8,
     show_progress_bar: bool = True,
-    auto_download_nltk: bool = True,
 ) -> pd.DataFrame:
     """
     Calculates RAG evaluation metrics from input dataframes.
@@ -261,7 +276,7 @@ def create_rag_eval_metrics_report(
         A Langchain chat model used for calculating generation metrics.
 
     metric_binds : List[MetricBind], optional, default=None
-        A list of string constants from `aidial_rag_eval.metric_binds`
+        A list of metric bind instances from `aidial_rag_eval.metric_binds`
         specifying the generation metrics.
 
     max_concurrency : int, default=8
@@ -287,6 +302,5 @@ def create_rag_eval_metrics_report(
         metric_binds,
         max_concurrency,
         show_progress_bar,
-        auto_download_nltk,
     )
     return pd.concat([df_merged, retrieval_metrics, generation_metrics], axis=1)

{aidial_rag_eval-0.6.0.dev16 → aidial_rag_eval-0.6.0.dev20}/src/aidial_rag_eval/evaluate.py

@@ -21,7 +21,6 @@ def evaluate(
     metric_binds: Optional[List[MetricBind]] = None,
     max_concurrency: int = 8,
     show_progress_bar: bool = True,
-    auto_download_nltk: bool = False,
 ) -> Dataset:
     """
     Calculates RAG evaluation metrics from input
@@ -83,7 +82,6 @@ def evaluate(
         metric_binds=metric_binds,
         max_concurrency=max_concurrency,
         show_progress_bar=show_progress_bar,
-        auto_download_nltk=auto_download_nltk,
     )
     aggregated_metrics = df_final.mean(numeric_only=True)
     assert isinstance(aggregated_metrics, pd.Series)

{aidial_rag_eval-0.6.0.dev16 → aidial_rag_eval-0.6.0.dev20}/src/aidial_rag_eval/generation/inference.py

@@ -30,13 +30,15 @@ from aidial_rag_eval.types import Documents, Question
 
 T = TypeVar("T")
 
+_LIST_DELIMITER = "\n"
+
 
 def _join_documents(documents: Documents) -> JoinedDocumentsName:
     return " ; ".join(documents)
 
 
 def _make_inference_task_inputs(
-    premises: List[Premise],
+    adjusted_premises: List[str],
     statements: List[Union[HypothesisStatements, ErrorInfo]],
     document_names: List[JoinedDocumentsName],
 ) -> List[InferenceInputs]:
@@ -45,7 +47,7 @@ def _make_inference_task_inputs(
 
     Parameters
     -----------
-    premises : List[str]
+    adjusted_premises : List[str]
         A list of premises from which we want to derive hypotheses in pairs.
 
     statements : List[Union[HypothesisStatements, ErrorInfo]]
@@ -73,7 +75,7 @@ def _make_inference_task_inputs(
                 [
                     InferenceInputs(
                         hypothesis_id=i,
-                        premise=premises[i],
+                        premise=adjusted_premises[i],
                         statements=list_statements,
                         document_name=document_names[i],
                     )
@@ -83,7 +85,7 @@ def _make_inference_task_inputs(
                 else [
                     InferenceInputs(
                         hypothesis_id=i,
-                        premise=premises[i],
+                        premise=adjusted_premises[i],
                         statements=[],
                         document_name=document_names[i],
                         error=statement_result,
@@ -203,7 +205,6 @@ def _segment_hypotheses(
     llm: BaseChatModel,
     max_concurrency: int = 8,
     show_progress_bar: bool = True,
-    auto_download_nltk: bool = True,
 ) -> Tuple[List[SegmentedText], List[Union[SegmentedText, ErrorInfo]]]:
     """
     Function that segments hypotheses into hypothesis segments(roughly into
@@ -212,8 +213,9 @@ def _segment_hypotheses(
     Parameters
     -----------
 
-    hypotheses : List[str]
+    hypotheses : List[Union[str, List[str]]]
         The text of the hypothesis.
+        Each hypothesis can be a single text or a list of text fragments.
 
     llm : BaseChatModel
         The Langchain chat model used for calculating inference.
@@ -236,7 +238,11 @@ def _segment_hypotheses(
         max_concurrency=max_concurrency,
     )
     segmented_hypotheses = [
-        SegmentedText.from_text(text=hypothesis, auto_download_nltk=auto_download_nltk)
+        (
+            SegmentedText.from_list(hypothesis, _LIST_DELIMITER)
+            if not isinstance(hypothesis, str)
+            else SegmentedText.from_text(hypothesis)
+        )
         for hypothesis in hypotheses
     ]
     if show_progress_bar:
@@ -291,18 +297,17 @@ def _extract_statements(
     return extractor.extract(segmented_hypotheses, show_progress_bar)
 
 
-def _add_questions_to_premises(
-    premises: List[Premise],
-    questions: Optional[List[Question]],
-    auto_download_nltk: bool,
-) -> List[Premise]:
-    adjusted_premises: List[Premise] = list(premises)
-    if questions is not None:
-        for i, question in enumerate(questions):
-            question_split = SegmentedText.from_text(
-                text=question, auto_download_nltk=auto_download_nltk
+def _prepend_premise_prefixes(
+    normalized_premises: List[str],
+    premise_prefixes: Optional[List[Question]],
+) -> List[str]:
+    adjusted_premises: List[str] = list(normalized_premises)
+    if premise_prefixes is not None:
+        for i, prefix in enumerate(premise_prefixes):
+            prefix_split = SegmentedText.from_text(text=prefix)
+            adjusted_premises[i] = _LIST_DELIMITER.join(
+                [prefix_split.segments[-1], normalized_premises[i]]
             )
-            adjusted_premises[i] = question_split.segments[-1] + "\n" + premises[i]
     return adjusted_premises
 
 
@@ -310,11 +315,10 @@ def _infer_statements(
     premises: List[Premise],
     statements: List[Union[HypothesisStatements, ErrorInfo]],
     llm: BaseChatModel,
-    questions: Optional[List[Question]] = None,
+    premise_prefixes: Optional[List[Question]] = None,
     list_documents: Optional[List[Documents]] = None,
     max_concurrency: int = 8,
     show_progress_bar: bool = True,
-    auto_download_nltk: bool = True,
 ) -> List[List[Tuple[InferenceInputs, InferenceScore]]]:
     """
     Function that infers statements.
@@ -325,8 +329,9 @@ def _infer_statements(
     Parameters
     -----------
 
-    premises : List[str]
+    premises : List[Union[str, List[str]]]
         The text of the premise from which the hypothesis will be inferred.
+        Each premise can be a single text or a list of text fragments.
 
     statements : List[Union[HypothesisStatements, ErrorInfo]]
         A deeply nested list of statements, where the outermost
@@ -358,9 +363,10 @@ def _infer_statements(
         For items that failed in any stage, InferenceScore will have inference=None
         and the error field set.
     """
-    adjusted_premises = _add_questions_to_premises(
-        premises, questions, auto_download_nltk
-    )
+    normalized_premises: List[str] = [
+        _LIST_DELIMITER.join(p) if not isinstance(p, str) else p for p in premises
+    ]
+    adjusted_premises = _prepend_premise_prefixes(normalized_premises, premise_prefixes)
 
     document_names: List[JoinedDocumentsName] = (
         [""] * len(statements)
@@ -391,14 +397,12 @@ def segment_hypotheses(
     llm: BaseChatModel,
     max_concurrency: int = 8,
     show_progress_bar: bool = True,
-    auto_download_nltk: bool = True,
 ) -> List[Union[SegmentedText, ErrorInfo]]:
     return _segment_hypotheses(
         hypotheses=hypotheses,
         llm=llm,
         max_concurrency=max_concurrency,
         show_progress_bar=show_progress_bar,
-        auto_download_nltk=auto_download_nltk,
     )[1]
 
 
@@ -422,21 +426,19 @@ def infer_statements(
     premises: List[Premise],
     statements: List[HypothesisStatements],
     llm: BaseChatModel,
-    questions: Optional[List[Question]] = None,
+    premise_prefixes: Optional[List[Question]] = None,
     list_documents: Optional[List[Documents]] = None,
     max_concurrency: int = 8,
     show_progress_bar: bool = True,
-    auto_download_nltk: bool = True,
 ) -> List[List[Tuple[InferenceInputs, InferenceScore]]]:
     return _infer_statements(
         premises=premises,
         statements=cast(List[Union[HypothesisStatements, ErrorInfo]], statements),
         llm=llm,
-        questions=questions,
+        premise_prefixes=premise_prefixes,
         list_documents=list_documents,
         max_concurrency=max_concurrency,
         show_progress_bar=show_progress_bar,
-        auto_download_nltk=auto_download_nltk,
     )
 
 
@@ -480,11 +482,10 @@ def calculate_batch_inference(
     premises: List[Premise],
     hypotheses: List[Hypothesis],
     llm: BaseChatModel,
-    questions: Optional[List[Question]] = None,
+    premise_prefixes: Optional[List[Question]] = None,
     list_documents: Optional[List[Documents]] = None,
     max_concurrency: int = 8,
     show_progress_bar: bool = True,
-    auto_download_nltk: bool = True,
 ) -> List[InferenceReturn]:
     """
     Calculates pairwise the inference of a hypotheses from a premises.
@@ -492,11 +493,13 @@ def calculate_batch_inference(
     Parameters
     -----------
 
-    premises : List[str]
+    premises : List[Union[str, List[str]]]
         The text of the premise from which the hypothesis will be inferred.
+        Each premise can be a single text or a list of text fragments.
 
-    hypotheses : List[str]
+    hypotheses : List[Union[str, List[str]]]
         The text of the hypothesis.
+        Each hypothesis can be a single text or a list of text fragments.
 
     llm : BaseChatModel
         The Langchain chat model used for calculating inference.
@@ -530,7 +533,6 @@ def calculate_batch_inference(
         llm=llm,
         max_concurrency=max_concurrency,
         show_progress_bar=show_progress_bar,
-        auto_download_nltk=auto_download_nltk,
     )
     statements: List[Union[HypothesisStatements, ErrorInfo]] = _extract_statements(
         segmented_hypotheses=decontextualized_segmented_hypotheses,
@@ -543,11 +545,10 @@ def calculate_batch_inference(
             premises=premises,
             statements=statements,
             llm=llm,
-            questions=questions,
+            premise_prefixes=premise_prefixes,
             list_documents=list_documents,
             max_concurrency=max_concurrency,
             show_progress_bar=show_progress_bar,
-            auto_download_nltk=auto_download_nltk,
         )
     )
 
@@ -581,11 +582,10 @@ def calculate_inference(
     premise: Premise,
     hypothesis: Hypothesis,
     llm: BaseChatModel,
-    question: Optional[Question] = None,
+    premise_prefix: Optional[Question] = None,
     documents: Optional[Documents] = None,
     max_concurrency: int = 8,
     show_progress_bar: bool = True,
-    auto_download_nltk: bool = True,
 ) -> InferenceReturn:
     """
     Calculates the inference of a hypothesis from a premise.
@@ -593,11 +593,13 @@ def calculate_inference(
     Parameters
     -----------
 
-    premise : str
+    premise : Union[str, List[str]]
         The text of the premise from which the hypothesis will be inferred.
+        Can be a single text or a list of text fragments.
 
-    hypothesis : str
+    hypothesis : Union[str, List[str]]
         The text of the hypothesis.
+        Can be a single text or a list of text fragments.
 
     llm : BaseChatModel
         The Langchain chat model used for calculating inference.
@@ -621,16 +623,15 @@ def calculate_inference(
         along with a JSON string that explains how the inference was derived and
         highlights string used for highlighting each segment of the hypothesis.
     """
-    questions = None if question is None else [question]
+    premise_prefixes = None if premise_prefix is None else [premise_prefix]
     list_documents = None if documents is None else [documents]
     inference_returns = calculate_batch_inference(
         premises=[premise],
         hypotheses=[hypothesis],
         llm=llm,
-        questions=questions,
+        premise_prefixes=premise_prefixes,
         list_documents=list_documents,
         max_concurrency=max_concurrency,
         show_progress_bar=show_progress_bar,
-        auto_download_nltk=auto_download_nltk,
     )
     return inference_returns[0]