spark-nlp 6.1.0__tar.gz → 6.1.1__tar.gz

{spark_nlp-6.1.0 → spark_nlp-6.1.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: spark-nlp
-Version: 6.1.0
+Version: 6.1.1
 Summary: John Snow Labs Spark NLP is a natural language processing library built on top of Apache Spark ML. It provides simple, performant & accurate NLP annotations for machine learning pipelines, that scale easily in a distributed environment.
 Home-page: https://github.com/JohnSnowLabs/spark-nlp
 Author: John Snow Labs
@@ -58,7 +58,7 @@ Dynamic: summary
 
 Spark NLP is a state-of-the-art Natural Language Processing library built on top of Apache Spark. It provides **simple**, **performant** & **accurate** NLP annotations for machine learning pipelines that **scale** easily in a distributed environment.
 
-Spark NLP comes with **83000+** pretrained **pipelines** and **models** in more than **200+** languages.
+Spark NLP comes with **100000+** pretrained **pipelines** and **models** in more than **200+** languages.
 It also offers tasks such as **Tokenization**, **Word Segmentation**, **Part-of-Speech Tagging**, Word and Sentence **Embeddings**, **Named Entity Recognition**, **Dependency Parsing**, **Spell Checking**, **Text Classification**, **Sentiment Analysis**, **Token Classification**, **Machine Translation** (+180 languages), **Summarization**, **Question Answering**, **Table Question Answering**, **Text Generation**, **Image Classification**, **Image to Text (captioning)**, **Automatic Speech Recognition**, **Zero-Shot Learning**, and many more [NLP tasks](#features).
 
 **Spark NLP** is the only open-source NLP library in **production** that offers state-of-the-art transformers such as **BERT**, **CamemBERT**, **ALBERT**, **ELECTRA**, **XLNet**, **DistilBERT**, **RoBERTa**, **DeBERTa**, **XLM-RoBERTa**, **Longformer**, **ELMO**, **Universal Sentence Encoder**, **Llama-2**, **M2M100**, **BART**, **Instructor**, **E5**, **Google T5**, **MarianMT**, **OpenAI GPT2**, **Vision Transformers (ViT)**, **OpenAI Whisper**, **Llama**, **Mistral**, **Phi**, **Qwen2**, and many more not only to **Python** and **R**, but also to **JVM** ecosystem (**Java**, **Scala**, and **Kotlin**) at **scale** by extending **Apache Spark** natively.
@@ -102,7 +102,7 @@ $ java -version
 $ conda create -n sparknlp python=3.7 -y
 $ conda activate sparknlp
 # spark-nlp by default is based on pyspark 3.x
-$ pip install spark-nlp==6.1.0 pyspark==3.3.1
+$ pip install spark-nlp==6.1.1 pyspark==3.3.1
 ```
 
 In Python console or Jupyter `Python3` kernel:
@@ -168,11 +168,11 @@ For a quick example of using pipelines and models take a look at our official [d
 
 ### Apache Spark Support
 
-Spark NLP *6.1.0* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
+Spark NLP *6.1.1* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
 
 | Spark NLP | Apache Spark 3.5.x | Apache Spark 3.4.x | Apache Spark 3.3.x | Apache Spark 3.2.x | Apache Spark 3.1.x | Apache Spark 3.0.x | Apache Spark 2.4.x | Apache Spark 2.3.x |
 |-----------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|
-| 6.0.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
+| 6.x.x and up     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.5.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.4.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.3.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
@@ -198,7 +198,7 @@ Find out more about 4.x `SparkNLP` versions in our official [documentation](http
 
 ### Databricks Support
 
-Spark NLP 6.1.0 has been tested and is compatible with the following runtimes:
+Spark NLP 6.1.1 has been tested and is compatible with the following runtimes:
 
 | **CPU**            | **GPU**            |
 |--------------------|--------------------|
@@ -206,16 +206,17 @@ Spark NLP 6.1.0 has been tested and is compatible with the following runtimes:
 | 14.2 / 14.2 ML     | 14.2 ML & GPU      |
 | 14.3 / 14.3 ML     | 14.3 ML & GPU      |
 | 15.0 / 15.0 ML     | 15.0 ML & GPU      |
-| 15.1 / 15.0 ML     | 15.1 ML & GPU      |
-| 15.2 / 15.0 ML     | 15.2 ML & GPU      |
-| 15.3 / 15.0 ML     | 15.3 ML & GPU      |
-| 15.4 / 15.0 ML     | 15.4 ML & GPU      |
+| 15.1 / 15.1 ML     | 15.1 ML & GPU      |
+| 15.2 / 15.2 ML     | 15.2 ML & GPU      |
+| 15.3 / 15.3 ML     | 15.3 ML & GPU      |
+| 15.4 / 15.4 ML     | 15.4 ML & GPU      |
+| 16.4 / 16.4 ML     | 16.4 ML & GPU      |
 
 We are compatible with older runtimes. For a full list check databricks support in our official [documentation](https://sparknlp.org/docs/en/install#databricks-support)
 
 ### EMR Support
 
-Spark NLP 6.1.0 has been tested and is compatible with the following EMR releases:
+Spark NLP 6.1.1 has been tested and is compatible with the following EMR releases:
 
 | **EMR Release**    |
 |--------------------|

{spark_nlp-6.1.0 → spark_nlp-6.1.1}/README.md

@@ -19,7 +19,7 @@
 
 Spark NLP is a state-of-the-art Natural Language Processing library built on top of Apache Spark. It provides **simple**, **performant** & **accurate** NLP annotations for machine learning pipelines that **scale** easily in a distributed environment.
 
-Spark NLP comes with **83000+** pretrained **pipelines** and **models** in more than **200+** languages.
+Spark NLP comes with **100000+** pretrained **pipelines** and **models** in more than **200+** languages.
 It also offers tasks such as **Tokenization**, **Word Segmentation**, **Part-of-Speech Tagging**, Word and Sentence **Embeddings**, **Named Entity Recognition**, **Dependency Parsing**, **Spell Checking**, **Text Classification**, **Sentiment Analysis**, **Token Classification**, **Machine Translation** (+180 languages), **Summarization**, **Question Answering**, **Table Question Answering**, **Text Generation**, **Image Classification**, **Image to Text (captioning)**, **Automatic Speech Recognition**, **Zero-Shot Learning**, and many more [NLP tasks](#features).
 
 **Spark NLP** is the only open-source NLP library in **production** that offers state-of-the-art transformers such as **BERT**, **CamemBERT**, **ALBERT**, **ELECTRA**, **XLNet**, **DistilBERT**, **RoBERTa**, **DeBERTa**, **XLM-RoBERTa**, **Longformer**, **ELMO**, **Universal Sentence Encoder**, **Llama-2**, **M2M100**, **BART**, **Instructor**, **E5**, **Google T5**, **MarianMT**, **OpenAI GPT2**, **Vision Transformers (ViT)**, **OpenAI Whisper**, **Llama**, **Mistral**, **Phi**, **Qwen2**, and many more not only to **Python** and **R**, but also to **JVM** ecosystem (**Java**, **Scala**, and **Kotlin**) at **scale** by extending **Apache Spark** natively.
@@ -63,7 +63,7 @@ $ java -version
 $ conda create -n sparknlp python=3.7 -y
 $ conda activate sparknlp
 # spark-nlp by default is based on pyspark 3.x
-$ pip install spark-nlp==6.1.0 pyspark==3.3.1
+$ pip install spark-nlp==6.1.1 pyspark==3.3.1
 ```
 
 In Python console or Jupyter `Python3` kernel:
@@ -129,11 +129,11 @@ For a quick example of using pipelines and models take a look at our official [d
 
 ### Apache Spark Support
 
-Spark NLP *6.1.0* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
+Spark NLP *6.1.1* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
 
 | Spark NLP | Apache Spark 3.5.x | Apache Spark 3.4.x | Apache Spark 3.3.x | Apache Spark 3.2.x | Apache Spark 3.1.x | Apache Spark 3.0.x | Apache Spark 2.4.x | Apache Spark 2.3.x |
 |-----------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|
-| 6.0.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
+| 6.x.x and up     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.5.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.4.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.3.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
@@ -159,7 +159,7 @@ Find out more about 4.x `SparkNLP` versions in our official [documentation](http
 
 ### Databricks Support
 
-Spark NLP 6.1.0 has been tested and is compatible with the following runtimes:
+Spark NLP 6.1.1 has been tested and is compatible with the following runtimes:
 
 | **CPU**            | **GPU**            |
 |--------------------|--------------------|
@@ -167,16 +167,17 @@ Spark NLP 6.1.0 has been tested and is compatible with the following runtimes:
 | 14.2 / 14.2 ML     | 14.2 ML & GPU      |
 | 14.3 / 14.3 ML     | 14.3 ML & GPU      |
 | 15.0 / 15.0 ML     | 15.0 ML & GPU      |
-| 15.1 / 15.0 ML     | 15.1 ML & GPU      |
-| 15.2 / 15.0 ML     | 15.2 ML & GPU      |
-| 15.3 / 15.0 ML     | 15.3 ML & GPU      |
-| 15.4 / 15.0 ML     | 15.4 ML & GPU      |
+| 15.1 / 15.1 ML     | 15.1 ML & GPU      |
+| 15.2 / 15.2 ML     | 15.2 ML & GPU      |
+| 15.3 / 15.3 ML     | 15.3 ML & GPU      |
+| 15.4 / 15.4 ML     | 15.4 ML & GPU      |
+| 16.4 / 16.4 ML     | 16.4 ML & GPU      |
 
 We are compatible with older runtimes. For a full list check databricks support in our official [documentation](https://sparknlp.org/docs/en/install#databricks-support)
 
 ### EMR Support
 
-Spark NLP 6.1.0 has been tested and is compatible with the following EMR releases:
+Spark NLP 6.1.1 has been tested and is compatible with the following EMR releases:
 
 | **EMR Release**    |
 |--------------------|

{spark_nlp-6.1.0 → spark_nlp-6.1.1}/setup.py

@@ -41,7 +41,7 @@ setup(
     # project code, see
     # https://packaging.python.org/en/latest/single_source_version.html
 
-    version='6.1.0',  # Required
+    version='6.1.1',  # Required
 
     # This is a one-line description or tagline of what your project does. This
     # corresponds to the 'Summary' metadata field:

{spark_nlp-6.1.0 → spark_nlp-6.1.1}/spark_nlp.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: spark-nlp
-Version: 6.1.0
+Version: 6.1.1
 Summary: John Snow Labs Spark NLP is a natural language processing library built on top of Apache Spark ML. It provides simple, performant & accurate NLP annotations for machine learning pipelines, that scale easily in a distributed environment.
 Home-page: https://github.com/JohnSnowLabs/spark-nlp
 Author: John Snow Labs
@@ -58,7 +58,7 @@ Dynamic: summary
 
 Spark NLP is a state-of-the-art Natural Language Processing library built on top of Apache Spark. It provides **simple**, **performant** & **accurate** NLP annotations for machine learning pipelines that **scale** easily in a distributed environment.
 
-Spark NLP comes with **83000+** pretrained **pipelines** and **models** in more than **200+** languages.
+Spark NLP comes with **100000+** pretrained **pipelines** and **models** in more than **200+** languages.
 It also offers tasks such as **Tokenization**, **Word Segmentation**, **Part-of-Speech Tagging**, Word and Sentence **Embeddings**, **Named Entity Recognition**, **Dependency Parsing**, **Spell Checking**, **Text Classification**, **Sentiment Analysis**, **Token Classification**, **Machine Translation** (+180 languages), **Summarization**, **Question Answering**, **Table Question Answering**, **Text Generation**, **Image Classification**, **Image to Text (captioning)**, **Automatic Speech Recognition**, **Zero-Shot Learning**, and many more [NLP tasks](#features).
 
 **Spark NLP** is the only open-source NLP library in **production** that offers state-of-the-art transformers such as **BERT**, **CamemBERT**, **ALBERT**, **ELECTRA**, **XLNet**, **DistilBERT**, **RoBERTa**, **DeBERTa**, **XLM-RoBERTa**, **Longformer**, **ELMO**, **Universal Sentence Encoder**, **Llama-2**, **M2M100**, **BART**, **Instructor**, **E5**, **Google T5**, **MarianMT**, **OpenAI GPT2**, **Vision Transformers (ViT)**, **OpenAI Whisper**, **Llama**, **Mistral**, **Phi**, **Qwen2**, and many more not only to **Python** and **R**, but also to **JVM** ecosystem (**Java**, **Scala**, and **Kotlin**) at **scale** by extending **Apache Spark** natively.
@@ -102,7 +102,7 @@ $ java -version
 $ conda create -n sparknlp python=3.7 -y
 $ conda activate sparknlp
 # spark-nlp by default is based on pyspark 3.x
-$ pip install spark-nlp==6.1.0 pyspark==3.3.1
+$ pip install spark-nlp==6.1.1 pyspark==3.3.1
 ```
 
 In Python console or Jupyter `Python3` kernel:
@@ -168,11 +168,11 @@ For a quick example of using pipelines and models take a look at our official [d
 
 ### Apache Spark Support
 
-Spark NLP *6.1.0* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
+Spark NLP *6.1.1* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
 
 | Spark NLP | Apache Spark 3.5.x | Apache Spark 3.4.x | Apache Spark 3.3.x | Apache Spark 3.2.x | Apache Spark 3.1.x | Apache Spark 3.0.x | Apache Spark 2.4.x | Apache Spark 2.3.x |
 |-----------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|
-| 6.0.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
+| 6.x.x and up     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.5.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.4.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.3.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
@@ -198,7 +198,7 @@ Find out more about 4.x `SparkNLP` versions in our official [documentation](http
 
 ### Databricks Support
 
-Spark NLP 6.1.0 has been tested and is compatible with the following runtimes:
+Spark NLP 6.1.1 has been tested and is compatible with the following runtimes:
 
 | **CPU**            | **GPU**            |
 |--------------------|--------------------|
@@ -206,16 +206,17 @@ Spark NLP 6.1.0 has been tested and is compatible with the following runtimes:
 | 14.2 / 14.2 ML     | 14.2 ML & GPU      |
 | 14.3 / 14.3 ML     | 14.3 ML & GPU      |
 | 15.0 / 15.0 ML     | 15.0 ML & GPU      |
-| 15.1 / 15.0 ML     | 15.1 ML & GPU      |
-| 15.2 / 15.0 ML     | 15.2 ML & GPU      |
-| 15.3 / 15.0 ML     | 15.3 ML & GPU      |
-| 15.4 / 15.0 ML     | 15.4 ML & GPU      |
+| 15.1 / 15.1 ML     | 15.1 ML & GPU      |
+| 15.2 / 15.2 ML     | 15.2 ML & GPU      |
+| 15.3 / 15.3 ML     | 15.3 ML & GPU      |
+| 15.4 / 15.4 ML     | 15.4 ML & GPU      |
+| 16.4 / 16.4 ML     | 16.4 ML & GPU      |
 
 We are compatible with older runtimes. For a full list check databricks support in our official [documentation](https://sparknlp.org/docs/en/install#databricks-support)
 
 ### EMR Support
 
-Spark NLP 6.1.0 has been tested and is compatible with the following EMR releases:
+Spark NLP 6.1.1 has been tested and is compatible with the following EMR releases:
 
 | **EMR Release**    |
 |--------------------|

{spark_nlp-6.1.0 → spark_nlp-6.1.1}/spark_nlp.egg-info/SOURCES.txt

@@ -255,6 +255,7 @@ sparknlp/reader/__init__.py
 sparknlp/reader/enums.py
 sparknlp/reader/pdf_to_text.py
 sparknlp/reader/reader2doc.py
+sparknlp/reader/reader2table.py
 sparknlp/reader/sparknlp_reader.py
 sparknlp/training/__init__.py
 sparknlp/training/conll.py

{spark_nlp-6.1.0 → spark_nlp-6.1.1}/sparknlp/__init__.py

@@ -66,7 +66,7 @@ sys.modules['com.johnsnowlabs.ml.ai'] = annotator
 annotators = annotator
 embeddings = annotator
 
-__version__ = "6.1.0"
+__version__ = "6.1.1"
 
 
 def start(gpu=False,

{spark_nlp-6.1.0 → spark_nlp-6.1.1}/sparknlp/annotator/embeddings/auto_gguf_embeddings.py

@@ -12,8 +12,6 @@
 #  See the License for the specific language governing permissions and
 #  limitations under the License.
 """Contains classes for the AutoGGUFEmbeddings."""
-from typing import List
-
 from sparknlp.common import *
 
 
@@ -32,7 +30,7 @@ class AutoGGUFEmbeddings(AnnotatorModel, HasBatchedAnnotate):
     ...     .setInputCols(["document"]) \\
     ...     .setOutputCol("embeddings")
 
-    The default model is ``"Nomic_Embed_Text_v1.5.Q8_0.gguf"``, if no name is provided.
+    The default model is ``"Qwen3_Embedding_0.6B_Q8_0_gguf"``, if no name is provided.
 
     For extended examples of usage, see the
     `AutoGGUFEmbeddingsTest <https://github.com/JohnSnowLabs/spark-nlp/tree/master/src/test/scala/com/johnsnowlabs/nlp/embeddings/AutoGGUFEmbeddingsTest.scala>`__
@@ -313,12 +311,6 @@ class AutoGGUFEmbeddings(AnnotatorModel, HasBatchedAnnotate):
         "Set the pooling type for embeddings, use model default if unspecified",
         typeConverter=TypeConverters.toString,
     )
-    embedding = Param(
-        Params._dummy(),
-        "embedding",
-        "Whether to load model with embedding support",
-        typeConverter=TypeConverters.toBoolean,
-    )
     flashAttention = Param(
         Params._dummy(),
         "flashAttention",
@@ -489,10 +481,10 @@ class AutoGGUFEmbeddings(AnnotatorModel, HasBatchedAnnotate):
             classname=classname, java_model=java_model
         )
         self._setDefault(
-            embedding=True,
             nCtx=4096,
             nBatch=512,
             poolingType="MEAN",
+            nGpuLayers=99,
         )
 
     @staticmethod
@@ -517,13 +509,13 @@ class AutoGGUFEmbeddings(AnnotatorModel, HasBatchedAnnotate):
         return AutoGGUFEmbeddings(java_model=jModel)
 
     @staticmethod
-    def pretrained(name="Nomic_Embed_Text_v1.5.Q8_0.gguf", lang="en", remote_loc=None):
+    def pretrained(name="Qwen3_Embedding_0.6B_Q8_0_gguf", lang="en", remote_loc=None):
         """Downloads and loads a pretrained model.
 
         Parameters
         ----------
         name : str, optional
-            Name of the pretrained model, by default "Nomic_Embed_Text_v1.5.Q8_0.gguf"
+            Name of the pretrained model, by default "Qwen3_Embedding_0.6B_Q8_0_gguf"
         lang : str, optional
             Language of the pretrained model, by default "en"
         remote_loc : str, optional

{spark_nlp-6.1.0 → spark_nlp-6.1.1}/sparknlp/annotator/seq2seq/auto_gguf_model.py

@@ -37,7 +37,11 @@ class AutoGGUFModel(AnnotatorModel, HasBatchedAnnotate, HasLlamaCppProperties):
     ...     .setInputCols(["document"]) \\
     ...     .setOutputCol("completions")
 
-    The default model is ``"phi3.5_mini_4k_instruct_q4_gguf"``, if no name is provided.
+    The default model is ``"Phi_4_mini_instruct_Q4_K_M_gguf"``, if no name is provided.
+
+    AutoGGUFModel is also able to load pretrained models from AutoGGUFVisionModel. Just
+    specify the same name for the pretrained method, and it will load the text-part of the
+    multimodal model automatically.
 
     For extended examples of usage, see the
     `AutoGGUFModelTest <https://github.com/JohnSnowLabs/spark-nlp/tree/master/src/test/scala/com/johnsnowlabs/nlp/annotators/seq2seq/AutoGGUFModelTest.scala>`__
@@ -120,8 +124,6 @@ class AutoGGUFModel(AnnotatorModel, HasBatchedAnnotate, HasLlamaCppProperties):
         Set path to static lookup cache to use for lookup decoding (not updated by generation)
     lookupCacheDynamicFilePath
         Set path to dynamic lookup cache to use for lookup decoding (updated by generation)
-    embedding
-        Whether to load model with embedding support
     flashAttention
         Whether to enable Flash Attention
     inputPrefixBos
@@ -252,7 +254,6 @@ class AutoGGUFModel(AnnotatorModel, HasBatchedAnnotate, HasLlamaCppProperties):
             useChatTemplate=True,
             nCtx=4096,
             nBatch=512,
-            embedding=False,
             nPredict=100,
             nGpuLayers=99,
             systemPrompt="You are a helpful assistant."
@@ -279,13 +280,13 @@ class AutoGGUFModel(AnnotatorModel, HasBatchedAnnotate, HasLlamaCppProperties):
         return AutoGGUFModel(java_model=jModel)
 
     @staticmethod
-    def pretrained(name="phi3.5_mini_4k_instruct_q4_gguf", lang="en", remote_loc=None):
+    def pretrained(name="Phi_4_mini_instruct_Q4_K_M_gguf", lang="en", remote_loc=None):
         """Downloads and loads a pretrained model.
 
         Parameters
         ----------
         name : str, optional
-            Name of the pretrained model, by default "phi3.5_mini_4k_instruct_q4_gguf"
+            Name of the pretrained model, by default "Phi_4_mini_instruct_Q4_K_M_gguf"
         lang : str, optional
             Language of the pretrained model, by default "en"
         remote_loc : str, optional

{spark_nlp-6.1.0 → spark_nlp-6.1.1}/sparknlp/annotator/seq2seq/auto_gguf_vision_model.py

@@ -43,7 +43,7 @@ class AutoGGUFVisionModel(AnnotatorModel, HasBatchedAnnotate, HasLlamaCppPropert
             .setOutputCol("completions")
 
 
-    The default model is ``"llava_v1.5_7b_Q4_0_gguf"``, if no name is provided.
+    The default model is ``"Qwen2.5_VL_3B_Instruct_Q4_K_M_gguf"``, if no name is provided.
 
     For available pretrained models please see the `Models Hub <https://sparknlp.org/models>`__.
 
@@ -311,13 +311,13 @@ class AutoGGUFVisionModel(AnnotatorModel, HasBatchedAnnotate, HasLlamaCppPropert
         return AutoGGUFVisionModel(java_model=jModel)
 
     @staticmethod
-    def pretrained(name="llava_v1.5_7b_Q4_0_gguf", lang="en", remote_loc=None):
+    def pretrained(name="Qwen2.5_VL_3B_Instruct_Q4_K_M_gguf", lang="en", remote_loc=None):
         """Downloads and loads a pretrained model.
 
         Parameters
         ----------
         name : str, optional
-            Name of the pretrained model, by default "llava_v1.5_7b_Q4_0_gguf"
+            Name of the pretrained model, by default "Qwen2.5_VL_3B_Instruct_Q4_K_M_gguf"
         lang : str, optional
             Language of the pretrained model, by default "en"
         remote_loc : str, optional

{spark_nlp-6.1.0 → spark_nlp-6.1.1}/sparknlp/common/properties.py

@@ -628,7 +628,6 @@ class HasGeneratorProperties:
                              "The number of sequences to return from the beam search.",
                              typeConverter=TypeConverters.toInt)
 
-
     def setTask(self, value):
         """Sets the transformer's task, e.g. ``summarize:``.
 
@@ -639,7 +638,6 @@ class HasGeneratorProperties:
         """
         return self._set(task=value)
 
-
     def setMinOutputLength(self, value):
         """Sets minimum length of the sequence to be generated.
 
@@ -650,7 +648,6 @@ class HasGeneratorProperties:
         """
         return self._set(minOutputLength=value)
 
-
     def setMaxOutputLength(self, value):
         """Sets maximum length of output text.
 
@@ -661,7 +658,6 @@ class HasGeneratorProperties:
         """
         return self._set(maxOutputLength=value)
 
-
     def setDoSample(self, value):
         """Sets whether or not to use sampling, use greedy decoding otherwise.
 
@@ -672,7 +668,6 @@ class HasGeneratorProperties:
         """
         return self._set(doSample=value)
 
-
     def setTemperature(self, value):
         """Sets the value used to module the next token probabilities.
 
@@ -683,7 +678,6 @@ class HasGeneratorProperties:
         """
         return self._set(temperature=value)
 
-
     def setTopK(self, value):
         """Sets the number of highest probability vocabulary tokens to keep for
         top-k-filtering.
@@ -695,7 +689,6 @@ class HasGeneratorProperties:
         """
         return self._set(topK=value)
 
-
     def setTopP(self, value):
         """Sets the top cumulative probability for vocabulary tokens.
 
@@ -709,7 +702,6 @@ class HasGeneratorProperties:
         """
         return self._set(topP=value)
 
-
     def setRepetitionPenalty(self, value):
         """Sets the parameter for repetition penalty. 1.0 means no penalty.
 
@@ -725,7 +717,6 @@ class HasGeneratorProperties:
         """
         return self._set(repetitionPenalty=value)
 
-
     def setNoRepeatNgramSize(self, value):
         """Sets size of n-grams that can only occur once.
 
@@ -738,7 +729,6 @@ class HasGeneratorProperties:
         """
         return self._set(noRepeatNgramSize=value)
 
-
     def setBeamSize(self, value):
         """Sets the number of beam size for beam search.
 
@@ -749,7 +739,6 @@ class HasGeneratorProperties:
         """
         return self._set(beamSize=value)
 
-
     def setNReturnSequences(self, value):
         """Sets the number of sequences to return from the beam search.
 
@@ -845,11 +834,10 @@ class HasLlamaCppProperties:
                             typeConverter=TypeConverters.toString)
     # Set the pooling type for embeddings, use model default if unspecified
     #
-    #   - 0 NONE: Don't use any pooling
-    #   - 1 MEAN: Mean Pooling
-    #   - 2 CLS: CLS Pooling
-    #   - 3 LAST: Last token pooling
-    #   - 4 RANK: For reranked models
+    #   - MEAN: Mean Pooling
+    #   - CLS: CLS Pooling
+    #   - LAST: Last token pooling
+    #   - RANK: For reranked models
     poolingType = Param(Params._dummy(), "poolingType",
                         "Set the pooling type for embeddings, use model default if unspecified",
                         typeConverter=TypeConverters.toString)
@@ -882,6 +870,10 @@ class HasLlamaCppProperties:
                          typeConverter=TypeConverters.toString)
     chatTemplate = Param(Params._dummy(), "chatTemplate", "The chat template to use",
                          typeConverter=TypeConverters.toString)
+    logVerbosity = Param(Params._dummy(), "logVerbosity", "Set the log verbosity level",
+                         typeConverter=TypeConverters.toInt)
+    disableLog = Param(Params._dummy(), "disableLog", "Whether to disable logging",
+                       typeConverter=TypeConverters.toBoolean)
 
     # -------- INFERENCE PARAMETERS --------
     inputPrefix = Param(Params._dummy(), "inputPrefix", "Set the prompt to start generation with",
@@ -1082,10 +1074,10 @@ class HasLlamaCppProperties:
         ropeScalingTypeUpper = ropeScalingType.upper()
         ropeScalingTypes = ["NONE", "LINEAR", "YARN"]
         if ropeScalingTypeUpper not in ropeScalingTypes:
-           raise ValueError(
-               f"Invalid RoPE scaling type: {ropeScalingType}. "
-               + f"Valid values are: {ropeScalingTypes}"
-           )
+            raise ValueError(
+                f"Invalid RoPE scaling type: {ropeScalingType}. "
+                + f"Valid values are: {ropeScalingTypes}"
+            )
         return self._set(ropeScalingType=ropeScalingTypeUpper)
 
     def setPoolingType(self, poolingType: str):
@@ -1093,11 +1085,10 @@ class HasLlamaCppProperties:
 
         Possible values:
 
-        - 0 NONE: Don't use any pooling
-        - 1 MEAN: Mean Pooling
-        - 2 CLS: CLS Pooling
-        - 3 LAST: Last token pooling
-        - 4 RANK: For reranked models
+        - MEAN: Mean Pooling
+        - CLS: CLS Pooling
+        - LAST: Last token pooling
+        - RANK: For reranked models
         """
         poolingTypeUpper = poolingType.upper()
         poolingTypes = ["NONE", "MEAN", "CLS", "LAST", "RANK"]
@@ -1124,10 +1115,6 @@ class HasLlamaCppProperties:
     #     """Set path to dynamic lookup cache to use for lookup decoding (updated by generation)"""
     #     return self._set(lookupCacheDynamicFilePath=lookupCacheDynamicFilePath)
 
-    def setEmbedding(self, embedding: bool):
-        """Whether to load model with embedding support"""
-        return self._set(embedding=embedding)
-
     def setFlashAttention(self, flashAttention: bool):
         """Whether to enable Flash Attention"""
         return self._set(flashAttention=flashAttention)
@@ -1280,11 +1267,19 @@ class HasLlamaCppProperties:
     def setUseChatTemplate(self, useChatTemplate: bool):
         """Set whether generate should apply a chat template"""
         return self._set(useChatTemplate=useChatTemplate)
-    
+
     def setNParallel(self, nParallel: int):
         """Sets the number of parallel processes for decoding. This is an alias for `setBatchSize`."""
         return self.setBatchSize(nParallel)
 
+    def setLogVerbosity(self, logVerbosity: int):
+        """Set the log verbosity level"""
+        return self._set(logVerbosity=logVerbosity)
+
+    def setDisableLog(self, disableLog: bool):
+        """Whether to disable logging"""
+        return self._set(disableLog=disableLog)
+
     # -------- JAVA SETTERS --------
     def setTokenIdBias(self, tokenIdBias: Dict[int, float]):
         """Set token id bias"""

{spark_nlp-6.1.0 → spark_nlp-6.1.1}/sparknlp/reader/reader2doc.py

@@ -25,7 +25,7 @@ class Reader2Doc(
     HasExcelReaderProperties,
     HasHTMLReaderProperties,
     HasPowerPointProperties,
-    HasTextReaderProperties,
+    HasTextReaderProperties
 ):
     """
     The Reader2Doc annotator allows you to use reading files more smoothly within existing
@@ -36,7 +36,7 @@ class Reader2Doc(
     output as a structured Spark DataFrame.
 
     Supported formats include:
-    
+
     - Plain text
     - HTML
     - Word (.doc/.docx)
@@ -77,42 +77,49 @@ class Reader2Doc(
         Params._dummy(),
         "contentPath",
         "contentPath path to files to read",
-        typeConverter=TypeConverters.toString,
+        typeConverter=TypeConverters.toString
     )
 
     outputCol = Param(
         Params._dummy(),
         "outputCol",
         "output column name",
-        typeConverter=TypeConverters.toString,
+        typeConverter=TypeConverters.toString
     )
 
     contentType = Param(
         Params._dummy(),
         "contentType",
         "Set the content type to load following MIME specification",
-        typeConverter=TypeConverters.toString,
+        typeConverter=TypeConverters.toString
     )
 
     explodeDocs = Param(
         Params._dummy(),
         "explodeDocs",
         "whether to explode the documents into separate rows",
-        typeConverter=TypeConverters.toBoolean,
+        typeConverter=TypeConverters.toBoolean
     )
 
     flattenOutput = Param(
         Params._dummy(),
         "flattenOutput",
         "If true, output is flattened to plain text with minimal metadata",
-        typeConverter=TypeConverters.toBoolean,
+        typeConverter=TypeConverters.toBoolean
     )
 
     titleThreshold = Param(
         Params._dummy(),
         "titleThreshold",
         "Minimum font size threshold for title detection in PDF docs",
-        typeConverter=TypeConverters.toFloat,
+        typeConverter=TypeConverters.toFloat
+    )
+
+    outputFormat = Param(
+        Params._dummy(),
+        "outputFormat",
+        "Output format for the table content. Options are 'plain-text' or 'html-table'. Default is 'json-table'.",
+        typeConverter=TypeConverters.toString
     )
 
     @keyword_only
@@ -126,7 +133,6 @@ class Reader2Doc(
             titleThreshold=18
         )
     @keyword_only
-
     def setParams(self):
         kwargs = self._input_kwargs
         return self._set(**kwargs)
@@ -192,3 +198,13 @@ class Reader2Doc(
             Minimum font size threshold for title detection in PDF docs
         """
         return self._set(titleThreshold=value)
+
+    def setOutputFormat(self, value):
+        """Sets the output format for the table content.
+
+        Parameters
+        ----------
+        value : str
+            Output format for the table content. Options are 'plain-text' or 'html-table'. Default is 'json-table'.
+        """
+        return self._set(outputFormat=value)