spark-nlp 6.1.0__py2.py3-none-any.whl → 6.1.2rc1__py2.py3-none-any.whl

{spark_nlp-6.1.0.dist-info → spark_nlp-6.1.2rc1.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: spark-nlp
-Version: 6.1.0
+Version: 6.1.2rc1
 Summary: John Snow Labs Spark NLP is a natural language processing library built on top of Apache Spark ML. It provides simple, performant & accurate NLP annotations for machine learning pipelines, that scale easily in a distributed environment.
 Home-page: https://github.com/JohnSnowLabs/spark-nlp
 Author: John Snow Labs
@@ -58,7 +58,7 @@ Dynamic: summary
 
 Spark NLP is a state-of-the-art Natural Language Processing library built on top of Apache Spark. It provides **simple**, **performant** & **accurate** NLP annotations for machine learning pipelines that **scale** easily in a distributed environment.
 
-Spark NLP comes with **83000+** pretrained **pipelines** and **models** in more than **200+** languages.
+Spark NLP comes with **100000+** pretrained **pipelines** and **models** in more than **200+** languages.
 It also offers tasks such as **Tokenization**, **Word Segmentation**, **Part-of-Speech Tagging**, Word and Sentence **Embeddings**, **Named Entity Recognition**, **Dependency Parsing**, **Spell Checking**, **Text Classification**, **Sentiment Analysis**, **Token Classification**, **Machine Translation** (+180 languages), **Summarization**, **Question Answering**, **Table Question Answering**, **Text Generation**, **Image Classification**, **Image to Text (captioning)**, **Automatic Speech Recognition**, **Zero-Shot Learning**, and many more [NLP tasks](#features).
 
 **Spark NLP** is the only open-source NLP library in **production** that offers state-of-the-art transformers such as **BERT**, **CamemBERT**, **ALBERT**, **ELECTRA**, **XLNet**, **DistilBERT**, **RoBERTa**, **DeBERTa**, **XLM-RoBERTa**, **Longformer**, **ELMO**, **Universal Sentence Encoder**, **Llama-2**, **M2M100**, **BART**, **Instructor**, **E5**, **Google T5**, **MarianMT**, **OpenAI GPT2**, **Vision Transformers (ViT)**, **OpenAI Whisper**, **Llama**, **Mistral**, **Phi**, **Qwen2**, and many more not only to **Python** and **R**, but also to **JVM** ecosystem (**Java**, **Scala**, and **Kotlin**) at **scale** by extending **Apache Spark** natively.
@@ -102,7 +102,7 @@ $ java -version
 $ conda create -n sparknlp python=3.7 -y
 $ conda activate sparknlp
 # spark-nlp by default is based on pyspark 3.x
-$ pip install spark-nlp==6.1.0 pyspark==3.3.1
+$ pip install spark-nlp==6.1.1 pyspark==3.3.1
 ```
 
 In Python console or Jupyter `Python3` kernel:
@@ -168,11 +168,11 @@ For a quick example of using pipelines and models take a look at our official [d
 
 ### Apache Spark Support
 
-Spark NLP *6.1.0* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
+Spark NLP *6.1.1* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
 
 | Spark NLP | Apache Spark 3.5.x | Apache Spark 3.4.x | Apache Spark 3.3.x | Apache Spark 3.2.x | Apache Spark 3.1.x | Apache Spark 3.0.x | Apache Spark 2.4.x | Apache Spark 2.3.x |
 |-----------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|
-| 6.0.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
+| 6.x.x and up     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.5.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.4.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.3.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
@@ -198,7 +198,7 @@ Find out more about 4.x `SparkNLP` versions in our official [documentation](http
 
 ### Databricks Support
 
-Spark NLP 6.1.0 has been tested and is compatible with the following runtimes:
+Spark NLP 6.1.1 has been tested and is compatible with the following runtimes:
 
 | **CPU**            | **GPU**            |
 |--------------------|--------------------|
@@ -206,16 +206,17 @@ Spark NLP 6.1.0 has been tested and is compatible with the following runtimes:
 | 14.2 / 14.2 ML     | 14.2 ML & GPU      |
 | 14.3 / 14.3 ML     | 14.3 ML & GPU      |
 | 15.0 / 15.0 ML     | 15.0 ML & GPU      |
-| 15.1 / 15.0 ML     | 15.1 ML & GPU      |
-| 15.2 / 15.0 ML     | 15.2 ML & GPU      |
-| 15.3 / 15.0 ML     | 15.3 ML & GPU      |
-| 15.4 / 15.0 ML     | 15.4 ML & GPU      |
+| 15.1 / 15.1 ML     | 15.1 ML & GPU      |
+| 15.2 / 15.2 ML     | 15.2 ML & GPU      |
+| 15.3 / 15.3 ML     | 15.3 ML & GPU      |
+| 15.4 / 15.4 ML     | 15.4 ML & GPU      |
+| 16.4 / 16.4 ML     | 16.4 ML & GPU      |
 
 We are compatible with older runtimes. For a full list check databricks support in our official [documentation](https://sparknlp.org/docs/en/install#databricks-support)
 
 ### EMR Support
 
-Spark NLP 6.1.0 has been tested and is compatible with the following EMR releases:
+Spark NLP 6.1.1 has been tested and is compatible with the following EMR releases:
 
 | **EMR Release**    |
 |--------------------|

{spark_nlp-6.1.0.dist-info → spark_nlp-6.1.2rc1.dist-info}/RECORD

@@ -3,7 +3,7 @@ com/johnsnowlabs/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,
 com/johnsnowlabs/ml/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
 com/johnsnowlabs/ml/ai/__init__.py,sha256=YQiK2M7U4d8y5irPy_HB8ae0mSpqS9583MH44pnKJXc,295
 com/johnsnowlabs/nlp/__init__.py,sha256=DPIVXtONO5xXyOk-HB0-sNiHAcco17NN13zPS_6Uw8c,294
-sparknlp/__init__.py,sha256=wxPbTrab8A3tELe8XRaGCfuZ-T8Dc8szbOHXH9ZgLIU,13814
+sparknlp/__init__.py,sha256=sEwJ50P2C-LGn-MZ8oI6jO6N0OutYSL3b6bCVkOo2wc,13814
 sparknlp/annotation.py,sha256=I5zOxG5vV2RfPZfqN9enT1i4mo6oBcn3Lrzs37QiOiA,5635
 sparknlp/annotation_audio.py,sha256=iRV_InSVhgvAwSRe9NTbUH9v6OGvTM-FPCpSAKVu0mE,1917
 sparknlp/annotation_image.py,sha256=xhCe8Ko-77XqWVuuYHFrjKqF6zPd8Z-RY_rmZXNwCXU,2547
@@ -105,7 +105,7 @@ sparknlp/annotator/dependency/dependency_parser.py,sha256=SxyvHPp8Hs1Xnm5X1nLTMi
 sparknlp/annotator/dependency/typed_dependency_parser.py,sha256=60vPdYkbFk9MPGegg3m9Uik9cMXpMZd8tBvXG39gNww,12456
 sparknlp/annotator/embeddings/__init__.py,sha256=Aw1oaP5DI0OS6259c0TEZZ6j3VFSvYFEerah5a-udVw,2528
 sparknlp/annotator/embeddings/albert_embeddings.py,sha256=6Rd1LIn8oFIpq_ALcJh-RUjPEO7Ht8wsHY6JHSFyMkw,9995
-sparknlp/annotator/embeddings/auto_gguf_embeddings.py,sha256=IlqkPGOH2lmZvxEyDSGX-G90DtTFOe2Rvujfbg5zvlU,20185
+sparknlp/annotator/embeddings/auto_gguf_embeddings.py,sha256=TRAYbhGS4K8uSpsScvDr6uD3lYdxMpCUjwDMhV_74rM,19977
 sparknlp/annotator/embeddings/bert_embeddings.py,sha256=HVUjkg56kBcpGZCo-fmPG5uatMDF3swW_lnbpy1SgSI,8463
 sparknlp/annotator/embeddings/bert_sentence_embeddings.py,sha256=NQy9KuXT9aKsTpYCR5RAeoFWI2YqEGorbdYrf_0KKmw,9148
 sparknlp/annotator/embeddings/bge_embeddings.py,sha256=ZGbxssjJFaSfbcgqAPV5hsu81SnC0obgCVNOoJkArDA,8105
@@ -168,8 +168,8 @@ sparknlp/annotator/sentiment/__init__.py,sha256=Lq3vKaZS1YATLMg0VNXSVtkWL5q5G9ta
 sparknlp/annotator/sentiment/sentiment_detector.py,sha256=m545NGU0Xzg_PO6_qIfpli1uZj7JQcyFgqe9R6wAPFI,8154
 sparknlp/annotator/sentiment/vivekn_sentiment.py,sha256=4rpXWDgzU6ddnbrSCp9VdLb2epCc9oZ3c6XcqxEw8nk,9655
 sparknlp/annotator/seq2seq/__init__.py,sha256=Aj43G1MuQE0mW7LakCWPjiTkIGl7iHPAnKIwT_DfdIM,1781
-sparknlp/annotator/seq2seq/auto_gguf_model.py,sha256=Oah_RvOy9YrvfnnMRMKOGJHnAMYxo0SeczBZsndM3kY,11638
-sparknlp/annotator/seq2seq/auto_gguf_vision_model.py,sha256=EYrm8EW7AMq3AoIKPe7Gp6ayBlFpWeg76AsAr4nanqU,15346
+sparknlp/annotator/seq2seq/auto_gguf_model.py,sha256=yhZQHMHfp88rQvLHTWyS-8imZrwqp-8RQQwnw6PmHfc,11749
+sparknlp/annotator/seq2seq/auto_gguf_vision_model.py,sha256=swBek2026dW6BOX5O9P8Uq41X2GC71VGW0ADFeUIvs0,15299
 sparknlp/annotator/seq2seq/bart_transformer.py,sha256=I1flM4yeCzEAKOdQllBC30XuedxVJ7ferkFhZ6gwEbE,18481
 sparknlp/annotator/seq2seq/cohere_transformer.py,sha256=43LZBVazZMgJRCsN7HaYjVYfJ5hRMV95QZyxMtXq-m4,13496
 sparknlp/annotator/seq2seq/cpm_transformer.py,sha256=0CnBFMlxMu0pD2QZMHyoGtIYgXqfUQm68vr6zEAa6Eg,13290
@@ -223,7 +223,7 @@ sparknlp/common/annotator_properties.py,sha256=7B1os7pBUfHo6b7IPQAXQ-nir0u3tQLzD
 sparknlp/common/annotator_type.py,sha256=ash2Ip1IOOiJamPVyy_XQj8Ja_DRHm0b9Vj4Ni75oKM,1225
 sparknlp/common/coverage_result.py,sha256=No4PSh1HSs3PyRI1zC47x65tWgfirqPI290icHQoXEI,823
 sparknlp/common/match_strategy.py,sha256=kt1MUPqU1wCwk5qCdYk6jubHbU-5yfAYxb9jjAOrdnY,1678
-sparknlp/common/properties.py,sha256=4jDyxr2IGWEuNlGtOoPzqdCF7oLAKGy1z6MtqxUVMug,52704
+sparknlp/common/properties.py,sha256=7eBxODxKmFQAgOtrxUH9ly4LugUlkNRVXNQcM60AUK4,53025
 sparknlp/common/read_as.py,sha256=imxPGwV7jr4Li_acbo0OAHHRGCBbYv-akzEGaBWEfcY,1226
 sparknlp/common/recursive_annotator_approach.py,sha256=vqugBw22cE3Ff7PIpRlnYFuOlchgL0nM26D8j-NdpqU,1449
 sparknlp/common/storage.py,sha256=D91H3p8EIjNspjqAYu6ephRpCUtdcAir4_PrAbkIQWE,4842
@@ -247,7 +247,8 @@ sparknlp/pretrained/utils.py,sha256=T1MrvW_DaWk_jcOjVLOea0NMFE9w8fe0ZT_5urZ_nEY,
 sparknlp/reader/__init__.py,sha256=-Toj3AIBki-zXPpV8ezFTI2LX1yP_rK2bhpoa8nBkTw,685
 sparknlp/reader/enums.py,sha256=MNGug9oJ1BBLM1Pbske13kAabalDzHa2kucF5xzFpHs,770
 sparknlp/reader/pdf_to_text.py,sha256=eWw-cwjosmcSZ9eHso0F5QQoeGBBnwsOhzhCXXvMjZA,7169
-sparknlp/reader/reader2doc.py,sha256=xahxkEuNM21mb0-MHQoYLtDF1cbAYrMTRpN1-u5K3ec,6587
+sparknlp/reader/reader2doc.py,sha256=LRqfaL9nidhlPkJIwTJo7SnGYmNNfOqwEdrsWYGEdnI,7146
+sparknlp/reader/reader2table.py,sha256=GC6Yz0gQ83S6XKOi329TUNQuAvLrBxysqDkDRZPvcYA,4759
 sparknlp/reader/sparknlp_reader.py,sha256=MJs8v_ECYaV1SOabI1L_2MkVYEDVImtwgbYypO7DJSY,20623
 sparknlp/training/__init__.py,sha256=qREi9u-5Vc2VjpL6-XZsyvu5jSEIdIhowW7_kKaqMqo,852
 sparknlp/training/conll.py,sha256=wKBiSTrjc6mjsl7Nyt6B8f4yXsDJkZb-sn8iOjix9cE,6961
@@ -279,7 +280,7 @@ sparknlp/training/_tf_graph_builders_1x/ner_dl/dataset_encoder.py,sha256=R4yHFN3
 sparknlp/training/_tf_graph_builders_1x/ner_dl/ner_model.py,sha256=EoCSdcIjqQ3wv13MAuuWrKV8wyVBP0SbOEW41omHlR0,23189
 sparknlp/training/_tf_graph_builders_1x/ner_dl/ner_model_saver.py,sha256=k5CQ7gKV6HZbZMB8cKLUJuZxoZWlP_DFWdZ--aIDwsc,2356
 sparknlp/training/_tf_graph_builders_1x/ner_dl/sentence_grouper.py,sha256=pAxjWhjazSX8Vg0MFqJiuRVw1IbnQNSs-8Xp26L4nko,870
-spark_nlp-6.1.0.dist-info/METADATA,sha256=MDLwobOveRxQL45CWF-NY26iHa3a7PijF9wntBXpeZE,19722
-spark_nlp-6.1.0.dist-info/WHEEL,sha256=JNWh1Fm1UdwIQV075glCn4MVuCRs0sotJIq-J6rbxCU,109
-spark_nlp-6.1.0.dist-info/top_level.txt,sha256=uuytur4pyMRw2H_txNY2ZkaucZHUs22QF8-R03ch_-E,13
-spark_nlp-6.1.0.dist-info/RECORD,,
+spark_nlp-6.1.2rc1.dist-info/METADATA,sha256=4qK5_LPihfkDmSrLBQgH38R_VE5lzDnsygPpOccUTdc,19777
+spark_nlp-6.1.2rc1.dist-info/WHEEL,sha256=JNWh1Fm1UdwIQV075glCn4MVuCRs0sotJIq-J6rbxCU,109
+spark_nlp-6.1.2rc1.dist-info/top_level.txt,sha256=uuytur4pyMRw2H_txNY2ZkaucZHUs22QF8-R03ch_-E,13
+spark_nlp-6.1.2rc1.dist-info/RECORD,,

sparknlp/__init__.py

@@ -66,7 +66,7 @@ sys.modules['com.johnsnowlabs.ml.ai'] = annotator
 annotators = annotator
 embeddings = annotator
 
-__version__ = "6.1.0"
+__version__ = "6.1.1"
 
 
 def start(gpu=False,

sparknlp/annotator/embeddings/auto_gguf_embeddings.py

@@ -12,8 +12,6 @@
 #  See the License for the specific language governing permissions and
 #  limitations under the License.
 """Contains classes for the AutoGGUFEmbeddings."""
-from typing import List
-
 from sparknlp.common import *
 
 
@@ -32,7 +30,7 @@ class AutoGGUFEmbeddings(AnnotatorModel, HasBatchedAnnotate):
     ...     .setInputCols(["document"]) \\
     ...     .setOutputCol("embeddings")
 
-    The default model is ``"Nomic_Embed_Text_v1.5.Q8_0.gguf"``, if no name is provided.
+    The default model is ``"Qwen3_Embedding_0.6B_Q8_0_gguf"``, if no name is provided.
 
     For extended examples of usage, see the
     `AutoGGUFEmbeddingsTest <https://github.com/JohnSnowLabs/spark-nlp/tree/master/src/test/scala/com/johnsnowlabs/nlp/embeddings/AutoGGUFEmbeddingsTest.scala>`__
@@ -313,12 +311,6 @@ class AutoGGUFEmbeddings(AnnotatorModel, HasBatchedAnnotate):
         "Set the pooling type for embeddings, use model default if unspecified",
         typeConverter=TypeConverters.toString,
     )
-    embedding = Param(
-        Params._dummy(),
-        "embedding",
-        "Whether to load model with embedding support",
-        typeConverter=TypeConverters.toBoolean,
-    )
     flashAttention = Param(
         Params._dummy(),
         "flashAttention",
@@ -489,10 +481,10 @@ class AutoGGUFEmbeddings(AnnotatorModel, HasBatchedAnnotate):
             classname=classname, java_model=java_model
         )
         self._setDefault(
-            embedding=True,
             nCtx=4096,
             nBatch=512,
             poolingType="MEAN",
+            nGpuLayers=99,
         )
 
     @staticmethod
@@ -517,13 +509,13 @@ class AutoGGUFEmbeddings(AnnotatorModel, HasBatchedAnnotate):
         return AutoGGUFEmbeddings(java_model=jModel)
 
     @staticmethod
-    def pretrained(name="Nomic_Embed_Text_v1.5.Q8_0.gguf", lang="en", remote_loc=None):
+    def pretrained(name="Qwen3_Embedding_0.6B_Q8_0_gguf", lang="en", remote_loc=None):
         """Downloads and loads a pretrained model.
 
         Parameters
         ----------
         name : str, optional
-            Name of the pretrained model, by default "Nomic_Embed_Text_v1.5.Q8_0.gguf"
+            Name of the pretrained model, by default "Qwen3_Embedding_0.6B_Q8_0_gguf"
         lang : str, optional
             Language of the pretrained model, by default "en"
         remote_loc : str, optional

sparknlp/annotator/seq2seq/auto_gguf_model.py

@@ -37,7 +37,11 @@ class AutoGGUFModel(AnnotatorModel, HasBatchedAnnotate, HasLlamaCppProperties):
     ...     .setInputCols(["document"]) \\
     ...     .setOutputCol("completions")
 
-    The default model is ``"phi3.5_mini_4k_instruct_q4_gguf"``, if no name is provided.
+    The default model is ``"Phi_4_mini_instruct_Q4_K_M_gguf"``, if no name is provided.
+
+    AutoGGUFModel is also able to load pretrained models from AutoGGUFVisionModel. Just
+    specify the same name for the pretrained method, and it will load the text-part of the
+    multimodal model automatically.
 
     For extended examples of usage, see the
     `AutoGGUFModelTest <https://github.com/JohnSnowLabs/spark-nlp/tree/master/src/test/scala/com/johnsnowlabs/nlp/annotators/seq2seq/AutoGGUFModelTest.scala>`__
@@ -120,8 +124,6 @@ class AutoGGUFModel(AnnotatorModel, HasBatchedAnnotate, HasLlamaCppProperties):
         Set path to static lookup cache to use for lookup decoding (not updated by generation)
     lookupCacheDynamicFilePath
         Set path to dynamic lookup cache to use for lookup decoding (updated by generation)
-    embedding
-        Whether to load model with embedding support
     flashAttention
         Whether to enable Flash Attention
     inputPrefixBos
@@ -252,20 +254,19 @@ class AutoGGUFModel(AnnotatorModel, HasBatchedAnnotate, HasLlamaCppProperties):
             useChatTemplate=True,
             nCtx=4096,
             nBatch=512,
-            embedding=False,
             nPredict=100,
             nGpuLayers=99,
             systemPrompt="You are a helpful assistant."
         )
 
     @staticmethod
-    def loadSavedModel(folder, spark_session):
+    def loadSavedModel(path, spark_session):
         """Loads a locally saved model.
 
         Parameters
         ----------
-        folder : str
-            Folder of the saved model
+        path : str
+            Path to the gguf model
         spark_session : pyspark.sql.SparkSession
             The current SparkSession
 
@@ -275,17 +276,17 @@ class AutoGGUFModel(AnnotatorModel, HasBatchedAnnotate, HasLlamaCppProperties):
             The restored model
         """
         from sparknlp.internal import _AutoGGUFLoader
-        jModel = _AutoGGUFLoader(folder, spark_session._jsparkSession)._java_obj
+        jModel = _AutoGGUFLoader(path, spark_session._jsparkSession)._java_obj
         return AutoGGUFModel(java_model=jModel)
 
     @staticmethod
-    def pretrained(name="phi3.5_mini_4k_instruct_q4_gguf", lang="en", remote_loc=None):
+    def pretrained(name="Phi_4_mini_instruct_Q4_K_M_gguf", lang="en", remote_loc=None):
         """Downloads and loads a pretrained model.
 
         Parameters
         ----------
         name : str, optional
-            Name of the pretrained model, by default "phi3.5_mini_4k_instruct_q4_gguf"
+            Name of the pretrained model, by default "Phi_4_mini_instruct_Q4_K_M_gguf"
         lang : str, optional
             Language of the pretrained model, by default "en"
         remote_loc : str, optional

sparknlp/annotator/seq2seq/auto_gguf_vision_model.py

@@ -43,7 +43,7 @@ class AutoGGUFVisionModel(AnnotatorModel, HasBatchedAnnotate, HasLlamaCppPropert
             .setOutputCol("completions")
 
 
-    The default model is ``"llava_v1.5_7b_Q4_0_gguf"``, if no name is provided.
+    The default model is ``"Qwen2.5_VL_3B_Instruct_Q4_K_M_gguf"``, if no name is provided.
 
     For available pretrained models please see the `Models Hub <https://sparknlp.org/models>`__.
 
@@ -116,8 +116,6 @@ class AutoGGUFVisionModel(AnnotatorModel, HasBatchedAnnotate, HasLlamaCppPropert
         Set optimization strategies that help on some NUMA systems (if available)
     ropeScalingType
         Set the RoPE frequency scaling method, defaults to linear unless specified by the model
-    poolingType
-        Set the pooling type for embeddings, use model default if unspecified
     modelDraft
         Set the draft model for speculative decoding
     modelAlias
@@ -126,8 +124,6 @@ class AutoGGUFVisionModel(AnnotatorModel, HasBatchedAnnotate, HasLlamaCppPropert
         Set path to static lookup cache to use for lookup decoding (not updated by generation)
     lookupCacheDynamicFilePath
         Set path to dynamic lookup cache to use for lookup decoding (updated by generation)
-    embedding
-        Whether to load model with embedding support
     flashAttention
         Whether to enable Flash Attention
     inputPrefixBos
@@ -284,8 +280,10 @@ class AutoGGUFVisionModel(AnnotatorModel, HasBatchedAnnotate, HasLlamaCppPropert
             useChatTemplate=True,
             nCtx=4096,
             nBatch=512,
-            embedding=False,
-            nPredict=100
+            nPredict=100,
+            nGpuLayers=99,
+            systemPrompt="You are a helpful assistant.",
+            batchSize=2,
         )
 
     @staticmethod
@@ -311,13 +309,13 @@ class AutoGGUFVisionModel(AnnotatorModel, HasBatchedAnnotate, HasLlamaCppPropert
         return AutoGGUFVisionModel(java_model=jModel)
 
     @staticmethod
-    def pretrained(name="llava_v1.5_7b_Q4_0_gguf", lang="en", remote_loc=None):
+    def pretrained(name="Qwen2.5_VL_3B_Instruct_Q4_K_M_gguf", lang="en", remote_loc=None):
         """Downloads and loads a pretrained model.
 
         Parameters
         ----------
         name : str, optional
-            Name of the pretrained model, by default "llava_v1.5_7b_Q4_0_gguf"
+            Name of the pretrained model, by default "Qwen2.5_VL_3B_Instruct_Q4_K_M_gguf"
         lang : str, optional
             Language of the pretrained model, by default "en"
         remote_loc : str, optional

sparknlp/common/properties.py

@@ -628,7 +628,6 @@ class HasGeneratorProperties:
                              "The number of sequences to return from the beam search.",
                              typeConverter=TypeConverters.toInt)
 
-
     def setTask(self, value):
         """Sets the transformer's task, e.g. ``summarize:``.
 
@@ -639,7 +638,6 @@ class HasGeneratorProperties:
         """
         return self._set(task=value)
 
-
     def setMinOutputLength(self, value):
         """Sets minimum length of the sequence to be generated.
 
@@ -650,7 +648,6 @@ class HasGeneratorProperties:
         """
         return self._set(minOutputLength=value)
 
-
     def setMaxOutputLength(self, value):
         """Sets maximum length of output text.
 
@@ -661,7 +658,6 @@ class HasGeneratorProperties:
         """
         return self._set(maxOutputLength=value)
 
-
     def setDoSample(self, value):
         """Sets whether or not to use sampling, use greedy decoding otherwise.
 
@@ -672,7 +668,6 @@ class HasGeneratorProperties:
         """
         return self._set(doSample=value)
 
-
     def setTemperature(self, value):
         """Sets the value used to module the next token probabilities.
 
@@ -683,7 +678,6 @@ class HasGeneratorProperties:
         """
         return self._set(temperature=value)
 
-
     def setTopK(self, value):
         """Sets the number of highest probability vocabulary tokens to keep for
         top-k-filtering.
@@ -695,7 +689,6 @@ class HasGeneratorProperties:
         """
         return self._set(topK=value)
 
-
     def setTopP(self, value):
         """Sets the top cumulative probability for vocabulary tokens.
 
@@ -709,7 +702,6 @@ class HasGeneratorProperties:
         """
         return self._set(topP=value)
 
-
     def setRepetitionPenalty(self, value):
         """Sets the parameter for repetition penalty. 1.0 means no penalty.
 
@@ -725,7 +717,6 @@ class HasGeneratorProperties:
         """
         return self._set(repetitionPenalty=value)
 
-
     def setNoRepeatNgramSize(self, value):
         """Sets size of n-grams that can only occur once.
 
@@ -738,7 +729,6 @@ class HasGeneratorProperties:
         """
         return self._set(noRepeatNgramSize=value)
 
-
     def setBeamSize(self, value):
         """Sets the number of beam size for beam search.
 
@@ -749,7 +739,6 @@ class HasGeneratorProperties:
         """
         return self._set(beamSize=value)
 
-
     def setNReturnSequences(self, value):
         """Sets the number of sequences to return from the beam search.
 
@@ -845,11 +834,10 @@ class HasLlamaCppProperties:
                             typeConverter=TypeConverters.toString)
     # Set the pooling type for embeddings, use model default if unspecified
     #
-    #   - 0 NONE: Don't use any pooling
-    #   - 1 MEAN: Mean Pooling
-    #   - 2 CLS: CLS Pooling
-    #   - 3 LAST: Last token pooling
-    #   - 4 RANK: For reranked models
+    #   - MEAN: Mean Pooling
+    #   - CLS: CLS Pooling
+    #   - LAST: Last token pooling
+    #   - RANK: For reranked models
     poolingType = Param(Params._dummy(), "poolingType",
                         "Set the pooling type for embeddings, use model default if unspecified",
                         typeConverter=TypeConverters.toString)
@@ -882,6 +870,10 @@ class HasLlamaCppProperties:
                          typeConverter=TypeConverters.toString)
     chatTemplate = Param(Params._dummy(), "chatTemplate", "The chat template to use",
                          typeConverter=TypeConverters.toString)
+    logVerbosity = Param(Params._dummy(), "logVerbosity", "Set the log verbosity level",
+                         typeConverter=TypeConverters.toInt)
+    disableLog = Param(Params._dummy(), "disableLog", "Whether to disable logging",
+                       typeConverter=TypeConverters.toBoolean)
 
     # -------- INFERENCE PARAMETERS --------
     inputPrefix = Param(Params._dummy(), "inputPrefix", "Set the prompt to start generation with",
@@ -1082,10 +1074,10 @@ class HasLlamaCppProperties:
         ropeScalingTypeUpper = ropeScalingType.upper()
         ropeScalingTypes = ["NONE", "LINEAR", "YARN"]
         if ropeScalingTypeUpper not in ropeScalingTypes:
-           raise ValueError(
-               f"Invalid RoPE scaling type: {ropeScalingType}. "
-               + f"Valid values are: {ropeScalingTypes}"
-           )
+            raise ValueError(
+                f"Invalid RoPE scaling type: {ropeScalingType}. "
+                + f"Valid values are: {ropeScalingTypes}"
+            )
         return self._set(ropeScalingType=ropeScalingTypeUpper)
 
     def setPoolingType(self, poolingType: str):
@@ -1093,11 +1085,10 @@ class HasLlamaCppProperties:
 
         Possible values:
 
-        - 0 NONE: Don't use any pooling
-        - 1 MEAN: Mean Pooling
-        - 2 CLS: CLS Pooling
-        - 3 LAST: Last token pooling
-        - 4 RANK: For reranked models
+        - MEAN: Mean Pooling
+        - CLS: CLS Pooling
+        - LAST: Last token pooling
+        - RANK: For reranked models
         """
         poolingTypeUpper = poolingType.upper()
         poolingTypes = ["NONE", "MEAN", "CLS", "LAST", "RANK"]
@@ -1124,10 +1115,6 @@ class HasLlamaCppProperties:
     #     """Set path to dynamic lookup cache to use for lookup decoding (updated by generation)"""
     #     return self._set(lookupCacheDynamicFilePath=lookupCacheDynamicFilePath)
 
-    def setEmbedding(self, embedding: bool):
-        """Whether to load model with embedding support"""
-        return self._set(embedding=embedding)
-
     def setFlashAttention(self, flashAttention: bool):
         """Whether to enable Flash Attention"""
         return self._set(flashAttention=flashAttention)
@@ -1280,11 +1267,19 @@ class HasLlamaCppProperties:
     def setUseChatTemplate(self, useChatTemplate: bool):
         """Set whether generate should apply a chat template"""
         return self._set(useChatTemplate=useChatTemplate)
-    
+
     def setNParallel(self, nParallel: int):
         """Sets the number of parallel processes for decoding. This is an alias for `setBatchSize`."""
         return self.setBatchSize(nParallel)
 
+    def setLogVerbosity(self, logVerbosity: int):
+        """Set the log verbosity level"""
+        return self._set(logVerbosity=logVerbosity)
+
+    def setDisableLog(self, disableLog: bool):
+        """Whether to disable logging"""
+        return self._set(disableLog=disableLog)
+
     # -------- JAVA SETTERS --------
     def setTokenIdBias(self, tokenIdBias: Dict[int, float]):
         """Set token id bias"""

sparknlp/reader/reader2doc.py

@@ -25,7 +25,7 @@ class Reader2Doc(
     HasExcelReaderProperties,
     HasHTMLReaderProperties,
     HasPowerPointProperties,
-    HasTextReaderProperties,
+    HasTextReaderProperties
 ):
     """
     The Reader2Doc annotator allows you to use reading files more smoothly within existing
@@ -36,7 +36,7 @@ class Reader2Doc(
     output as a structured Spark DataFrame.
 
     Supported formats include:
-    
+
     - Plain text
     - HTML
     - Word (.doc/.docx)
@@ -77,42 +77,49 @@ class Reader2Doc(
         Params._dummy(),
         "contentPath",
         "contentPath path to files to read",
-        typeConverter=TypeConverters.toString,
+        typeConverter=TypeConverters.toString
     )
 
     outputCol = Param(
         Params._dummy(),
         "outputCol",
         "output column name",
-        typeConverter=TypeConverters.toString,
+        typeConverter=TypeConverters.toString
     )
 
     contentType = Param(
         Params._dummy(),
         "contentType",
         "Set the content type to load following MIME specification",
-        typeConverter=TypeConverters.toString,
+        typeConverter=TypeConverters.toString
     )
 
     explodeDocs = Param(
         Params._dummy(),
         "explodeDocs",
         "whether to explode the documents into separate rows",
-        typeConverter=TypeConverters.toBoolean,
+        typeConverter=TypeConverters.toBoolean
     )
 
     flattenOutput = Param(
         Params._dummy(),
         "flattenOutput",
         "If true, output is flattened to plain text with minimal metadata",
-        typeConverter=TypeConverters.toBoolean,
+        typeConverter=TypeConverters.toBoolean
     )
 
     titleThreshold = Param(
         Params._dummy(),
         "titleThreshold",
         "Minimum font size threshold for title detection in PDF docs",
-        typeConverter=TypeConverters.toFloat,
+        typeConverter=TypeConverters.toFloat
+    )
+
+    outputFormat = Param(
+        Params._dummy(),
+        "outputFormat",
+        "Output format for the table content. Options are 'plain-text' or 'html-table'. Default is 'json-table'.",
+        typeConverter=TypeConverters.toString
     )
 
     @keyword_only
@@ -126,7 +133,6 @@ class Reader2Doc(
             titleThreshold=18
         )
     @keyword_only
-
     def setParams(self):
         kwargs = self._input_kwargs
         return self._set(**kwargs)
@@ -192,3 +198,13 @@ class Reader2Doc(
             Minimum font size threshold for title detection in PDF docs
         """
         return self._set(titleThreshold=value)
+
+    def setOutputFormat(self, value):
+        """Sets the output format for the table content.
+
+        Parameters
+        ----------
+        value : str
+            Output format for the table content. Options are 'plain-text' or 'html-table'. Default is 'json-table'.
+        """
+        return self._set(outputFormat=value)

sparknlp/reader/reader2table.py

@@ -0,0 +1,163 @@
+#  Copyright 2017-2025 John Snow Labs
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+from pyspark import keyword_only
+from pyspark.ml.param import TypeConverters, Params, Param
+
+from sparknlp.common import AnnotatorType
+from sparknlp.internal import AnnotatorTransformer
+from sparknlp.partition.partition_properties import *
+
+class Reader2Table(
+    AnnotatorTransformer,
+    HasEmailReaderProperties,
+    HasExcelReaderProperties,
+    HasHTMLReaderProperties,
+    HasPowerPointProperties,
+    HasTextReaderProperties
+):
+    name = 'Reader2Table'
+
+    outputAnnotatorType = AnnotatorType.DOCUMENT
+
+    contentPath = Param(
+        Params._dummy(),
+        "contentPath",
+        "contentPath path to files to read",
+        typeConverter=TypeConverters.toString
+    )
+
+    outputCol = Param(
+        Params._dummy(),
+        "outputCol",
+        "output column name",
+        typeConverter=TypeConverters.toString
+    )
+
+    contentType = Param(
+        Params._dummy(),
+        "contentType",
+        "Set the content type to load following MIME specification",
+        typeConverter=TypeConverters.toString
+    )
+
+    explodeDocs = Param(
+        Params._dummy(),
+        "explodeDocs",
+        "whether to explode the documents into separate rows",
+        typeConverter=TypeConverters.toBoolean
+    )
+
+    flattenOutput = Param(
+        Params._dummy(),
+        "flattenOutput",
+        "If true, output is flattened to plain text with minimal metadata",
+        typeConverter=TypeConverters.toBoolean
+    )
+
+    titleThreshold = Param(
+        Params._dummy(),
+        "titleThreshold",
+        "Minimum font size threshold for title detection in PDF docs",
+        typeConverter=TypeConverters.toFloat
+    )
+
+    outputFormat = Param(
+        Params._dummy(),
+        "outputFormat",
+        "Output format for the table content. Options are 'plain-text' or 'html-table'. Default is 'json-table'.",
+        typeConverter=TypeConverters.toString
+    )
+
+    @keyword_only
+    def __init__(self):
+        super(Reader2Table, self).__init__(classname="com.johnsnowlabs.reader.Reader2Table")
+        self._setDefault(outputCol="document")
+
+    @keyword_only
+    def setParams(self):
+        kwargs = self._input_kwargs
+        return self._set(**kwargs)
+
+    def setContentPath(self, value):
+        """Sets content path.
+
+        Parameters
+        ----------
+        value : str
+            contentPath path to files to read
+        """
+        return self._set(contentPath=value)
+
+    def setContentType(self, value):
+        """
+        Set the content type to load following MIME specification
+
+        Parameters
+        ----------
+        value : str
+            content type to load following MIME specification
+        """
+        return self._set(contentType=value)
+
+    def setExplodeDocs(self, value):
+        """Sets whether to explode the documents into separate rows.
+
+
+        Parameters
+        ----------
+        value : boolean
+        Whether to explode the documents into separate rows
+        """
+        return self._set(explodeDocs=value)
+
+    def setOutputCol(self, value):
+        """Sets output column name.
+
+        Parameters
+        ----------
+        value : str
+            Name of the Output Column
+        """
+        return self._set(outputCol=value)
+
+    def setFlattenOutput(self, value):
+        """Sets whether to flatten the output to plain text with minimal metadata.
+
+        Parameters
+        ----------
+        value : bool
+            If true, output is flattened to plain text with minimal metadata
+        """
+        return self._set(flattenOutput=value)
+
+    def setTitleThreshold(self, value):
+        """Sets the minimum font size threshold for title detection in PDF documents.
+
+        Parameters
+        ----------
+        value : float
+            Minimum font size threshold for title detection in PDF docs
+        """
+        return self._set(titleThreshold=value)
+
+    def setOutputFormat(self, value):
+        """Sets the output format for the table content.
+
+        Parameters
+        ----------
+        value : str
+            Output format for the table content. Options are 'plain-text' or 'html-table'. Default is 'json-table'.
+        """
+        return self._set(outputFormat=value)