spark-nlp 6.0.2__tar.gz → 6.0.4__tar.gz

{spark_nlp-6.0.2 → spark_nlp-6.0.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: spark-nlp
-Version: 6.0.2
+Version: 6.0.4
 Summary: John Snow Labs Spark NLP is a natural language processing library built on top of Apache Spark ML. It provides simple, performant & accurate NLP annotations for machine learning pipelines, that scale easily in a distributed environment.
 Home-page: https://github.com/JohnSnowLabs/spark-nlp
 Author: John Snow Labs
@@ -102,7 +102,7 @@ $ java -version
 $ conda create -n sparknlp python=3.7 -y
 $ conda activate sparknlp
 # spark-nlp by default is based on pyspark 3.x
-$ pip install spark-nlp==6.0.2 pyspark==3.3.1
+$ pip install spark-nlp==6.0.4 pyspark==3.3.1
 ```
 
 In Python console or Jupyter `Python3` kernel:
@@ -168,7 +168,7 @@ For a quick example of using pipelines and models take a look at our official [d
 
 ### Apache Spark Support
 
-Spark NLP *6.0.2* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
+Spark NLP *6.0.4* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
 
 | Spark NLP | Apache Spark 3.5.x | Apache Spark 3.4.x | Apache Spark 3.3.x | Apache Spark 3.2.x | Apache Spark 3.1.x | Apache Spark 3.0.x | Apache Spark 2.4.x | Apache Spark 2.3.x |
 |-----------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|
@@ -198,7 +198,7 @@ Find out more about 4.x `SparkNLP` versions in our official [documentation](http
 
 ### Databricks Support
 
-Spark NLP 6.0.2 has been tested and is compatible with the following runtimes:
+Spark NLP 6.0.4 has been tested and is compatible with the following runtimes:
 
 | **CPU**            | **GPU**            |
 |--------------------|--------------------|
@@ -215,7 +215,7 @@ We are compatible with older runtimes. For a full list check databricks support
 
 ### EMR Support
 
-Spark NLP 6.0.2 has been tested and is compatible with the following EMR releases:
+Spark NLP 6.0.4 has been tested and is compatible with the following EMR releases:
 
 | **EMR Release**    |
 |--------------------|

{spark_nlp-6.0.2 → spark_nlp-6.0.4}/README.md

@@ -63,7 +63,7 @@ $ java -version
 $ conda create -n sparknlp python=3.7 -y
 $ conda activate sparknlp
 # spark-nlp by default is based on pyspark 3.x
-$ pip install spark-nlp==6.0.2 pyspark==3.3.1
+$ pip install spark-nlp==6.0.4 pyspark==3.3.1
 ```
 
 In Python console or Jupyter `Python3` kernel:
@@ -129,7 +129,7 @@ For a quick example of using pipelines and models take a look at our official [d
 
 ### Apache Spark Support
 
-Spark NLP *6.0.2* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
+Spark NLP *6.0.4* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
 
 | Spark NLP | Apache Spark 3.5.x | Apache Spark 3.4.x | Apache Spark 3.3.x | Apache Spark 3.2.x | Apache Spark 3.1.x | Apache Spark 3.0.x | Apache Spark 2.4.x | Apache Spark 2.3.x |
 |-----------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|
@@ -159,7 +159,7 @@ Find out more about 4.x `SparkNLP` versions in our official [documentation](http
 
 ### Databricks Support
 
-Spark NLP 6.0.2 has been tested and is compatible with the following runtimes:
+Spark NLP 6.0.4 has been tested and is compatible with the following runtimes:
 
 | **CPU**            | **GPU**            |
 |--------------------|--------------------|
@@ -176,7 +176,7 @@ We are compatible with older runtimes. For a full list check databricks support
 
 ### EMR Support
 
-Spark NLP 6.0.2 has been tested and is compatible with the following EMR releases:
+Spark NLP 6.0.4 has been tested and is compatible with the following EMR releases:
 
 | **EMR Release**    |
 |--------------------|

{spark_nlp-6.0.2 → spark_nlp-6.0.4}/setup.py

@@ -41,7 +41,7 @@ setup(
     # project code, see
     # https://packaging.python.org/en/latest/single_source_version.html
 
-    version='6.0.2',  # Required
+    version='6.0.4',  # Required
 
     # This is a one-line description or tagline of what your project does. This
     # corresponds to the 'Summary' metadata field:

{spark_nlp-6.0.2 → spark_nlp-6.0.4}/spark_nlp.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: spark-nlp
-Version: 6.0.2
+Version: 6.0.4
 Summary: John Snow Labs Spark NLP is a natural language processing library built on top of Apache Spark ML. It provides simple, performant & accurate NLP annotations for machine learning pipelines, that scale easily in a distributed environment.
 Home-page: https://github.com/JohnSnowLabs/spark-nlp
 Author: John Snow Labs
@@ -102,7 +102,7 @@ $ java -version
 $ conda create -n sparknlp python=3.7 -y
 $ conda activate sparknlp
 # spark-nlp by default is based on pyspark 3.x
-$ pip install spark-nlp==6.0.2 pyspark==3.3.1
+$ pip install spark-nlp==6.0.4 pyspark==3.3.1
 ```
 
 In Python console or Jupyter `Python3` kernel:
@@ -168,7 +168,7 @@ For a quick example of using pipelines and models take a look at our official [d
 
 ### Apache Spark Support
 
-Spark NLP *6.0.2* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
+Spark NLP *6.0.4* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
 
 | Spark NLP | Apache Spark 3.5.x | Apache Spark 3.4.x | Apache Spark 3.3.x | Apache Spark 3.2.x | Apache Spark 3.1.x | Apache Spark 3.0.x | Apache Spark 2.4.x | Apache Spark 2.3.x |
 |-----------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|
@@ -198,7 +198,7 @@ Find out more about 4.x `SparkNLP` versions in our official [documentation](http
 
 ### Databricks Support
 
-Spark NLP 6.0.2 has been tested and is compatible with the following runtimes:
+Spark NLP 6.0.4 has been tested and is compatible with the following runtimes:
 
 | **CPU**            | **GPU**            |
 |--------------------|--------------------|
@@ -215,7 +215,7 @@ We are compatible with older runtimes. For a full list check databricks support
 
 ### EMR Support
 
-Spark NLP 6.0.2 has been tested and is compatible with the following EMR releases:
+Spark NLP 6.0.4 has been tested and is compatible with the following EMR releases:
 
 | **EMR Release**    |
 |--------------------|

{spark_nlp-6.0.2 → spark_nlp-6.0.4}/spark_nlp.egg-info/SOURCES.txt

@@ -20,6 +20,7 @@ sparknlp/util.py
 sparknlp/annotator/__init__.py
 sparknlp/annotator/chunk2_doc.py
 sparknlp/annotator/chunker.py
+sparknlp/annotator/dataframe_optimizer.py
 sparknlp/annotator/date2_chunk.py
 sparknlp/annotator/document_character_text_splitter.py
 sparknlp/annotator/document_normalizer.py
@@ -121,9 +122,11 @@ sparknlp/annotator/embeddings/deberta_embeddings.py
 sparknlp/annotator/embeddings/distil_bert_embeddings.py
 sparknlp/annotator/embeddings/doc2vec.py
 sparknlp/annotator/embeddings/e5_embeddings.py
+sparknlp/annotator/embeddings/e5v_embeddings.py
 sparknlp/annotator/embeddings/elmo_embeddings.py
 sparknlp/annotator/embeddings/instructor_embeddings.py
 sparknlp/annotator/embeddings/longformer_embeddings.py
+sparknlp/annotator/embeddings/minilm_embeddings.py
 sparknlp/annotator/embeddings/mpnet_embeddings.py
 sparknlp/annotator/embeddings/mxbai_embeddings.py
 sparknlp/annotator/embeddings/nomic_embeddings.py

{spark_nlp-6.0.2 → spark_nlp-6.0.4}/sparknlp/__init__.py

@@ -66,7 +66,7 @@ sys.modules['com.johnsnowlabs.ml.ai'] = annotator
 annotators = annotator
 embeddings = annotator
 
-__version__ = "6.0.2"
+__version__ = "6.0.4"
 
 
 def start(gpu=False,

spark_nlp-6.0.4/sparknlp/annotator/dataframe_optimizer.py

@@ -0,0 +1,216 @@
+#  Copyright 2017-2025 John Snow Labs
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+from pyspark.ml import Transformer
+from pyspark.ml.param.shared import *
+from pyspark.sql import DataFrame
+from typing import Any
+
+# Custom converter for string-to-string dictionaries
+def toStringDict(value):
+    if not isinstance(value, dict):
+        raise TypeError("Expected a dictionary of strings.")
+    return {str(k): str(v) for k, v in value.items()}
+
+class DataFrameOptimizer(Transformer):
+    """
+    Optimizes a Spark DataFrame by repartitioning, optionally caching, and persisting it to disk.
+
+    This transformer is intended to improve performance for Spark NLP pipelines or when preparing
+    data for export. It allows partition tuning via `numPartitions` directly, or indirectly using
+    `executorCores` and `numWorkers`. The DataFrame can also be persisted in a specified format
+    (`csv`, `json`, or `parquet`) with additional writer options.
+
+    Parameters
+    ----------
+    executorCores : int, optional
+        Number of cores per Spark executor (used to compute number of partitions if `numPartitions` is not set).
+
+    numWorkers : int, optional
+        Number of executor nodes (used to compute number of partitions if `numPartitions` is not set).
+
+    numPartitions : int, optional
+        Target number of partitions for the DataFrame (overrides calculation via cores × workers).
+
+    doCache : bool, default False
+        Whether to cache the DataFrame after repartitioning.
+
+    persistPath : str, optional
+        Path to save the DataFrame output (if persistence is enabled).
+
+    persistFormat : str, optional
+        Format to persist the DataFrame in: one of `'csv'`, `'json'`, or `'parquet'`.
+
+    outputOptions : dict, optional
+        Dictionary of options for the DataFrameWriter (e.g., `{"compression": "snappy"}` for parquet).
+
+    Examples
+    --------
+    >>> optimizer = DataFrameOptimizer() \\
+    ...     .setExecutorCores(4) \\
+    ...     .setNumWorkers(5) \\
+    ...     .setDoCache(True) \\
+    ...     .setPersistPath("/tmp/out") \\
+    ...     .setPersistFormat("parquet") \\
+    ...     .setOutputOptions({"compression": "snappy"})
+
+    >>> optimized_df = optimizer.transform(input_df)
+
+    Notes
+    -----
+    - You must specify either `numPartitions`, or both `executorCores` and `numWorkers`.
+    - Schema is preserved; no columns are modified or removed.
+    """
+
+    executorCores = Param(
+        Params._dummy(),
+        "executorCores",
+        "Number of cores per executor",
+        typeConverter = TypeConverters.toInt
+    )
+    numWorkers = Param(
+        Params._dummy(),
+        "numWorkers",
+        "Number of Spark workers",
+        typeConverter = TypeConverters.toInt
+    )
+    numPartitions = Param(
+        Params._dummy(),
+        "numPartitions",
+        "Total number of partitions (overrides executorCores * numWorkers)",
+        typeConverter = TypeConverters.toInt
+    )
+    doCache = Param(
+        Params._dummy(),
+        "doCache",
+        "Whether to cache the DataFrame",
+        typeConverter = TypeConverters.toBoolean
+    )
+
+    persistPath = Param(
+        Params._dummy(),
+        "persistPath",
+        "Optional path to persist the DataFrame",
+        typeConverter = TypeConverters.toString
+    )
+    persistFormat = Param(
+        Params._dummy(),
+        "persistFormat",
+        "Format to persist: parquet, json, csv",
+        typeConverter = TypeConverters.toString
+    )
+
+    outputOptions = Param(
+        Params._dummy(),
+        "outputOptions",
+        "Additional writer options",
+        typeConverter=toStringDict
+    )
+
+    def __init__(self):
+        super().__init__()
+        self._setDefault(
+            doCache=False,
+            persistFormat="none",
+            numPartitions=1,
+            executorCores=1,
+            numWorkers=1
+        )
+
+    # Parameter setters
+    def setExecutorCores(self, value: int):
+        """Set the number of executor cores."""
+        return self._set(executorCores=value)
+
+    def setNumWorkers(self, value: int):
+        """Set the number of Spark workers."""
+        return self._set(numWorkers=value)
+
+    def setNumPartitions(self, value: int):
+        """Set the total number of partitions (overrides cores * workers)."""
+        return self._set(numPartitions=value)
+
+    def setDoCache(self, value: bool):
+        """Set whether to cache the DataFrame."""
+        return self._set(doCache=value)
+
+    def setPersistPath(self, value: str):
+        """Set the path where the DataFrame should be persisted."""
+        return self._set(persistPath=value)
+
+    def setPersistFormat(self, value: str):
+        """Set the format to persist the DataFrame (parquet, json, csv)."""
+        return self._set(persistFormat=value)
+
+    def setOutputOptions(self, value: dict):
+        """Set additional writer options (e.g. for csv headers)."""
+        return self._set(outputOptions=value)
+
+    # Optional bulk setter
+    def setParams(self, **kwargs: Any):
+        for param, value in kwargs.items():
+            self._set(**{param: value})
+        return self
+
+    def _transform(self, dataset: DataFrame) -> DataFrame:
+        self._validate_params()
+        part_count = self.getOrDefault(self.numPartitions)
+        cores = self.getOrDefault(self.executorCores)
+        workers = self.getOrDefault(self.numWorkers)
+        if cores is None or workers is None:
+            raise ValueError("Provide either numPartitions or both executorCores and numWorkers")
+        if part_count == 1:
+            part_count = cores * workers
+
+        optimized_df = dataset.repartition(part_count)
+
+        if self.getOrDefault(self.doCache):
+            optimized_df = optimized_df.cache()
+
+        format = self.getOrDefault(self.persistFormat).lower()
+        if format != "none":
+            path = self.getOrDefault(self.persistPath)
+            if not path:
+                raise ValueError("persistPath must be set when persistFormat is not 'none'")
+            writer = optimized_df.write.mode("overwrite")
+            if self.isDefined(self.outputOptions):
+                writer = writer.options(**self.getOrDefault(self.outputOptions))
+            if format == "parquet":
+                writer.parquet(path)
+            elif format == "json":
+                writer.json(path)
+            elif format == "csv":
+                writer.csv(path)
+            else:
+                raise ValueError(f"Unsupported format: {format}")
+
+        return optimized_df
+
+    def _validate_params(self):
+        if self.isDefined(self.executorCores):
+            val = self.getOrDefault(self.executorCores)
+            if val <= 0:
+                raise ValueError("executorCores must be > 0")
+
+        if self.isDefined(self.numWorkers):
+            val = self.getOrDefault(self.numWorkers)
+            if val <= 0:
+                raise ValueError("numWorkers must be > 0")
+
+        if self.isDefined(self.numPartitions):
+            val = self.getOrDefault(self.numPartitions)
+            if val <= 0:
+                raise ValueError("numPartitions must be > 0")
+
+        if self.isDefined(self.persistPath) and not self.isDefined(self.persistFormat):
+            raise ValueError("persistFormat must be defined when persistPath is set")

{spark_nlp-6.0.2 → spark_nlp-6.0.4}/sparknlp/annotator/embeddings/__init__.py

@@ -25,6 +25,7 @@ from sparknlp.annotator.embeddings.elmo_embeddings import *
 from sparknlp.annotator.embeddings.e5_embeddings import *
 from sparknlp.annotator.embeddings.instructor_embeddings import *
 from sparknlp.annotator.embeddings.longformer_embeddings import *
+from sparknlp.annotator.embeddings.minilm_embeddings import *
 from sparknlp.annotator.embeddings.mpnet_embeddings import *
 from sparknlp.annotator.embeddings.roberta_embeddings import *
 from sparknlp.annotator.embeddings.roberta_sentence_embeddings import *
@@ -41,3 +42,4 @@ from sparknlp.annotator.embeddings.mxbai_embeddings import *
 from sparknlp.annotator.embeddings.snowflake_embeddings import *
 from sparknlp.annotator.embeddings.nomic_embeddings import *
 from sparknlp.annotator.embeddings.auto_gguf_embeddings import *
+from sparknlp.annotator.embeddings.e5v_embeddings import *

spark_nlp-6.0.4/sparknlp/annotator/embeddings/e5v_embeddings.py

@@ -0,0 +1,138 @@
+#  Copyright 2017-2024 John Snow Labs
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+from sparknlp.common import *
+
+class E5VEmbeddings(AnnotatorModel,
+                   HasBatchedAnnotateImage,
+                   HasImageFeatureProperties,
+                   HasEngine,
+                    HasRescaleFactor):
+    """Universal multimodal embeddings using the E5-V model (see https://huggingface.co/royokong/e5-v).
+
+    E5-V bridges the modality gap between different input types (text, image) and demonstrates strong performance in multimodal embeddings, even without fine-tuning. It also supports a single-modality training approach, where the model is trained exclusively on text pairs, often yielding better performance than multimodal training.
+
+    Pretrained models can be loaded with :meth:`.pretrained` of the companion object:
+
+    >>> e5vEmbeddings = E5VEmbeddings.pretrained() \
+    ...     .setInputCols(["image_assembler"]) \
+    ...     .setOutputCol("e5v")
+
+    The default model is ``"e5v_int4"``, if no name is provided.
+
+    For available pretrained models please see the `Models Hub <https://sparknlp.org/models?task=Question+Answering>`__.
+
+    ====================== ======================
+    Input Annotation types Output Annotation type
+    ====================== ======================
+    ``IMAGE``              ``SENTENCE_EMBEDDINGS``
+    ====================== ======================
+
+    Examples
+    --------
+    Image + Text Embedding:
+    >>> import sparknlp
+    >>> from sparknlp.base import *
+    >>> from sparknlp.annotator import *
+    >>> from pyspark.ml import Pipeline
+    >>> image_df = spark.read.format("image").option("dropInvalid", value = True).load(imageFolder)
+    >>> imagePrompt = "<|start_header_id|>user<|end_header_id|>\n\n<image>\\nSummary above image in one word: <|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n \n"
+    >>> test_df = image_df.withColumn("text", lit(imagePrompt))
+    >>> imageAssembler = ImageAssembler() \
+    ...     .setInputCol("image") \
+    ...     .setOutputCol("image_assembler")
+    >>> e5vEmbeddings = E5VEmbeddings.pretrained() \
+    ...     .setInputCols(["image_assembler"]) \
+    ...     .setOutputCol("e5v")
+    >>> pipeline = Pipeline().setStages([
+    ...     imageAssembler,
+    ...     e5vEmbeddings
+    ... ])
+    >>> result = pipeline.fit(test_df).transform(test_df)
+    >>> result.select("e5v.embeddings").show(truncate = False)
+
+    Text-Only Embedding:
+    >>> from sparknlp.util import EmbeddingsDataFrameUtils
+    >>> textPrompt = "<|start_header_id|>user<|end_header_id|>\n\n<sent>\\nSummary above sentence in one word: <|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n \n"
+    >>> textDesc = "A cat sitting in a box."
+    >>> nullImageDF = spark.createDataFrame(spark.sparkContext.parallelize([EmbeddingsDataFrameUtils.emptyImageRow]), EmbeddingsDataFrameUtils.imageSchema)
+    >>> textDF = nullImageDF.withColumn("text", lit(textPrompt.replace("<sent>", textDesc)))
+    >>> e5vEmbeddings = E5VEmbeddings.pretrained() \
+    ...     .setInputCols(["image"]) \
+    ...     .setOutputCol("e5v")
+    >>> result = e5vEmbeddings.transform(textDF)
+    >>> result.select("e5v.embeddings").show(truncate = False)
+    """
+
+    name = "E5VEmbeddings"
+
+    inputAnnotatorTypes = [AnnotatorType.IMAGE]
+    outputAnnotatorType = AnnotatorType.SENTENCE_EMBEDDINGS
+
+    @keyword_only
+    def __init__(self, classname="com.johnsnowlabs.nlp.embeddings.E5VEmbeddings", java_model=None):
+        """Initializes the E5VEmbeddings annotator.
+
+        Parameters
+        ----------
+        classname : str, optional
+            The Java class name of the annotator, by default "com.johnsnowlabs.nlp.annotators.embeddings.E5VEmbeddings"
+        java_model : Optional[java.lang.Object], optional
+            A pre-initialized Java model, by default None
+        """
+        super(E5VEmbeddings, self).__init__(classname=classname, java_model=java_model)
+        self._setDefault()
+
+    @staticmethod
+    def loadSavedModel(folder, spark_session, use_openvino=False):
+        """Loads a locally saved model.
+
+        Parameters
+        ----------
+        folder : str
+            Folder of the saved model
+        spark_session : pyspark.sql.SparkSession
+            The current SparkSession
+        use_openvino : bool, optional
+            Whether to use OpenVINO engine, by default False
+
+        Returns
+        -------
+        E5VEmbeddings
+            The restored model
+        """
+        from sparknlp.internal import _E5VEmbeddingsLoader
+        jModel = _E5VEmbeddingsLoader(folder, spark_session._jsparkSession, use_openvino)._java_obj
+        return E5VEmbeddings(java_model=jModel)
+
+    @staticmethod
+    def pretrained(name="e5v_int4", lang="en", remote_loc=None):
+        """Downloads and loads a pretrained model.
+
+        Parameters
+        ----------
+        name : str, optional
+            Name of the pretrained model, by default "e5v_int4"
+        lang : str, optional
+            Language of the pretrained model, by default "en"
+        remote_loc : str, optional
+            Optional remote address of the resource, by default None. Will use Spark NLPs repositories otherwise.
+
+        Returns
+        -------
+        E5VEmbeddings
+            The restored model
+        """
+        from sparknlp.pretrained import ResourceDownloader
+        return ResourceDownloader.downloadModel(E5VEmbeddings, name, lang, remote_loc)