spark-nlp 5.5.3__tar.gz → 6.0.1__tar.gz

{spark-nlp-5.5.3 → spark_nlp-6.0.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: spark-nlp
-Version: 5.5.3
+Version: 6.0.1
 Summary: John Snow Labs Spark NLP is a natural language processing library built on top of Apache Spark ML. It provides simple, performant & accurate NLP annotations for machine learning pipelines, that scale easily in a distributed environment.
 Home-page: https://github.com/JohnSnowLabs/spark-nlp
 Author: John Snow Labs
@@ -87,7 +87,7 @@ documentation and examples
 
 ## Quick Start
 
-This is a quick example of how to use Spark NLP pre-trained pipeline in Python and PySpark:
+This is a quick example of how to use a Spark NLP pre-trained pipeline in Python and PySpark:
 
 ```sh
 $ java -version
@@ -95,7 +95,7 @@ $ java -version
 $ conda create -n sparknlp python=3.7 -y
 $ conda activate sparknlp
 # spark-nlp by default is based on pyspark 3.x
-$ pip install spark-nlp==5.5.3 pyspark==3.3.1
+$ pip install spark-nlp==6.0.1 pyspark==3.3.1
 ```
 
 In Python console or Jupyter `Python3` kernel:
@@ -161,10 +161,11 @@ For a quick example of using pipelines and models take a look at our official [d
 
 ### Apache Spark Support
 
-Spark NLP *5.5.3* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
+Spark NLP *6.0.1* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
 
 | Spark NLP | Apache Spark 3.5.x | Apache Spark 3.4.x | Apache Spark 3.3.x | Apache Spark 3.2.x | Apache Spark 3.1.x | Apache Spark 3.0.x | Apache Spark 2.4.x | Apache Spark 2.3.x |
 |-----------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|
+| 6.0.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.5.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.4.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.3.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
@@ -178,6 +179,7 @@ Find out more about `Spark NLP` versions from our [release notes](https://github
 
 | Spark NLP | Python 3.6 | Python 3.7 | Python 3.8 | Python 3.9 | Python 3.10| Scala 2.11 | Scala 2.12 |
 |-----------|------------|------------|------------|------------|------------|------------|------------|
+| 6.0.x     | NO         | YES        | YES        | YES        | YES        | NO         | YES        |
 | 5.5.x     | NO         | YES        | YES        | YES        | YES        | NO         | YES        |
 | 5.4.x     | NO         | YES        | YES        | YES        | YES        | NO         | YES        |
 | 5.3.x     | NO         | YES        | YES        | YES        | YES        | NO         | YES        |
@@ -189,7 +191,7 @@ Find out more about 4.x `SparkNLP` versions in our official [documentation](http
 
 ### Databricks Support
 
-Spark NLP 5.5.3 has been tested and is compatible with the following runtimes:
+Spark NLP 6.0.1 has been tested and is compatible with the following runtimes:
 
 | **CPU**            | **GPU**            |
 |--------------------|--------------------|
@@ -206,7 +208,7 @@ We are compatible with older runtimes. For a full list check databricks support
 
 ### EMR Support
 
-Spark NLP 5.5.3 has been tested and is compatible with the following EMR releases:
+Spark NLP 6.0.1 has been tested and is compatible with the following EMR releases:
 
 | **EMR Release**    |
 |--------------------|
@@ -216,6 +218,13 @@ Spark NLP 5.5.3 has been tested and is compatible with the following EMR release
 | emr-7.0.0          |
 | emr-7.1.0          |
 | emr-7.2.0          |
+| emr-7.3.0          |
+| emr-7.4.0          |
+| emr-7.5.0          |
+| emr-7.6.0          |
+| emr-7.7.0          |
+| emr-7.8.0          |
+
 
 We are compatible with older EMR releases. For a full list check EMR support in our official [documentation](https://sparknlp.org/docs/en/install#emr-support)
 
@@ -237,7 +246,7 @@ deployed to Maven central. To add any of our packages as a dependency in your ap
 from our official documentation.
 
 If you are interested, there is a simple SBT project for Spark NLP to guide you on how to use it in your
-projects [Spark NLP SBT S5.5.3r](https://github.com/maziyarpanahi/spark-nlp-starter)
+projects [Spark NLP Starter](https://github.com/maziyarpanahi/spark-nlp-starter)
 
 ### Python
 
@@ -246,7 +255,7 @@ Check all available installations for Python in our official [documentation](htt
 
 ### Compiled JARs
 
-To compile the jars from source follow [these instructions](https://sparknlp.org/docs/en/compiled#jars) from our official documenation
+To compile the jars from source follow [these instructions](https://sparknlp.org/docs/en/compiled#jars) from our official documentation
 
 ## Platform-Specific Instructions
 
@@ -266,7 +275,7 @@ For detailed instructions on how to use Spark NLP on supported platforms, please
 
 Spark NLP library and all the pre-trained models/pipelines can be used entirely offline with no access to the Internet.
 Please check [these instructions](https://sparknlp.org/docs/en/install#s3-integration) from our official documentation
-to use Spark NLP offline
+to use Spark NLP offline.
 
 ## Advanced Settings
 
@@ -282,7 +291,7 @@ In Spark NLP we can define S3 locations to:
 
 Please check [these instructions](https://sparknlp.org/docs/en/install#s3-integration) from our official documentation.
 
-## Document5.5.3
+## Documentation
 
 ### Examples
 
@@ -315,7 +324,7 @@ the Spark NLP library:
     keywords = {Spark, Natural language processing, Deep learning, Tensorflow, Cluster},
     abstract = {Spark NLP is a Natural Language Processing (NLP) library built on top of Apache Spark ML. It provides simple, performant & accurate NLP annotations for machine learning pipelines that can scale easily in a distributed environment. Spark NLP comes with 1100+ pretrained pipelines and models in more than 192+ languages. It supports nearly all the NLP tasks and modules that can be used seamlessly in a cluster. Downloaded more than 2.7 million times and experiencing 9x growth since January 2020, Spark NLP is used by 54% of healthcare organizations as the world’s most widely used NLP library in the enterprise.}
     }
-}5.5.3
+}
 ```
 
 ## Community support

{spark-nlp-5.5.3 → spark_nlp-6.0.1}/README.md

@@ -55,7 +55,7 @@ documentation and examples
 
 ## Quick Start
 
-This is a quick example of how to use Spark NLP pre-trained pipeline in Python and PySpark:
+This is a quick example of how to use a Spark NLP pre-trained pipeline in Python and PySpark:
 
 ```sh
 $ java -version
@@ -63,7 +63,7 @@ $ java -version
 $ conda create -n sparknlp python=3.7 -y
 $ conda activate sparknlp
 # spark-nlp by default is based on pyspark 3.x
-$ pip install spark-nlp==5.5.3 pyspark==3.3.1
+$ pip install spark-nlp==6.0.1 pyspark==3.3.1
 ```
 
 In Python console or Jupyter `Python3` kernel:
@@ -129,10 +129,11 @@ For a quick example of using pipelines and models take a look at our official [d
 
 ### Apache Spark Support
 
-Spark NLP *5.5.3* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
+Spark NLP *6.0.1* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
 
 | Spark NLP | Apache Spark 3.5.x | Apache Spark 3.4.x | Apache Spark 3.3.x | Apache Spark 3.2.x | Apache Spark 3.1.x | Apache Spark 3.0.x | Apache Spark 2.4.x | Apache Spark 2.3.x |
 |-----------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|
+| 6.0.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.5.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.4.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.3.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
@@ -146,6 +147,7 @@ Find out more about `Spark NLP` versions from our [release notes](https://github
 
 | Spark NLP | Python 3.6 | Python 3.7 | Python 3.8 | Python 3.9 | Python 3.10| Scala 2.11 | Scala 2.12 |
 |-----------|------------|------------|------------|------------|------------|------------|------------|
+| 6.0.x     | NO         | YES        | YES        | YES        | YES        | NO         | YES        |
 | 5.5.x     | NO         | YES        | YES        | YES        | YES        | NO         | YES        |
 | 5.4.x     | NO         | YES        | YES        | YES        | YES        | NO         | YES        |
 | 5.3.x     | NO         | YES        | YES        | YES        | YES        | NO         | YES        |
@@ -157,7 +159,7 @@ Find out more about 4.x `SparkNLP` versions in our official [documentation](http
 
 ### Databricks Support
 
-Spark NLP 5.5.3 has been tested and is compatible with the following runtimes:
+Spark NLP 6.0.1 has been tested and is compatible with the following runtimes:
 
 | **CPU**            | **GPU**            |
 |--------------------|--------------------|
@@ -174,7 +176,7 @@ We are compatible with older runtimes. For a full list check databricks support
 
 ### EMR Support
 
-Spark NLP 5.5.3 has been tested and is compatible with the following EMR releases:
+Spark NLP 6.0.1 has been tested and is compatible with the following EMR releases:
 
 | **EMR Release**    |
 |--------------------|
@@ -184,6 +186,13 @@ Spark NLP 5.5.3 has been tested and is compatible with the following EMR release
 | emr-7.0.0          |
 | emr-7.1.0          |
 | emr-7.2.0          |
+| emr-7.3.0          |
+| emr-7.4.0          |
+| emr-7.5.0          |
+| emr-7.6.0          |
+| emr-7.7.0          |
+| emr-7.8.0          |
+
 
 We are compatible with older EMR releases. For a full list check EMR support in our official [documentation](https://sparknlp.org/docs/en/install#emr-support)
 
@@ -205,7 +214,7 @@ deployed to Maven central. To add any of our packages as a dependency in your ap
 from our official documentation.
 
 If you are interested, there is a simple SBT project for Spark NLP to guide you on how to use it in your
-projects [Spark NLP SBT S5.5.3r](https://github.com/maziyarpanahi/spark-nlp-starter)
+projects [Spark NLP Starter](https://github.com/maziyarpanahi/spark-nlp-starter)
 
 ### Python
 
@@ -214,7 +223,7 @@ Check all available installations for Python in our official [documentation](htt
 
 ### Compiled JARs
 
-To compile the jars from source follow [these instructions](https://sparknlp.org/docs/en/compiled#jars) from our official documenation
+To compile the jars from source follow [these instructions](https://sparknlp.org/docs/en/compiled#jars) from our official documentation
 
 ## Platform-Specific Instructions
 
@@ -234,7 +243,7 @@ For detailed instructions on how to use Spark NLP on supported platforms, please
 
 Spark NLP library and all the pre-trained models/pipelines can be used entirely offline with no access to the Internet.
 Please check [these instructions](https://sparknlp.org/docs/en/install#s3-integration) from our official documentation
-to use Spark NLP offline
+to use Spark NLP offline.
 
 ## Advanced Settings
 
@@ -250,7 +259,7 @@ In Spark NLP we can define S3 locations to:
 
 Please check [these instructions](https://sparknlp.org/docs/en/install#s3-integration) from our official documentation.
 
-## Document5.5.3
+## Documentation
 
 ### Examples
 
@@ -283,7 +292,7 @@ the Spark NLP library:
     keywords = {Spark, Natural language processing, Deep learning, Tensorflow, Cluster},
     abstract = {Spark NLP is a Natural Language Processing (NLP) library built on top of Apache Spark ML. It provides simple, performant & accurate NLP annotations for machine learning pipelines that can scale easily in a distributed environment. Spark NLP comes with 1100+ pretrained pipelines and models in more than 192+ languages. It supports nearly all the NLP tasks and modules that can be used seamlessly in a cluster. Downloaded more than 2.7 million times and experiencing 9x growth since January 2020, Spark NLP is used by 54% of healthcare organizations as the world’s most widely used NLP library in the enterprise.}
     }
-}5.5.3
+}
 ```
 
 ## Community support

{spark-nlp-5.5.3 → spark_nlp-6.0.1}/setup.py

@@ -41,7 +41,7 @@ setup(
     # project code, see
     # https://packaging.python.org/en/latest/single_source_version.html
 
-    version='5.5.3',  # Required
+    version='6.0.1',  # Required
 
     # This is a one-line description or tagline of what your project does. This
     # corresponds to the 'Summary' metadata field:

{spark-nlp-5.5.3 → spark_nlp-6.0.1}/spark_nlp.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: spark-nlp
-Version: 5.5.3
+Version: 6.0.1
 Summary: John Snow Labs Spark NLP is a natural language processing library built on top of Apache Spark ML. It provides simple, performant & accurate NLP annotations for machine learning pipelines, that scale easily in a distributed environment.
 Home-page: https://github.com/JohnSnowLabs/spark-nlp
 Author: John Snow Labs
@@ -87,7 +87,7 @@ documentation and examples
 
 ## Quick Start
 
-This is a quick example of how to use Spark NLP pre-trained pipeline in Python and PySpark:
+This is a quick example of how to use a Spark NLP pre-trained pipeline in Python and PySpark:
 
 ```sh
 $ java -version
@@ -95,7 +95,7 @@ $ java -version
 $ conda create -n sparknlp python=3.7 -y
 $ conda activate sparknlp
 # spark-nlp by default is based on pyspark 3.x
-$ pip install spark-nlp==5.5.3 pyspark==3.3.1
+$ pip install spark-nlp==6.0.1 pyspark==3.3.1
 ```
 
 In Python console or Jupyter `Python3` kernel:
@@ -161,10 +161,11 @@ For a quick example of using pipelines and models take a look at our official [d
 
 ### Apache Spark Support
 
-Spark NLP *5.5.3* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
+Spark NLP *6.0.1* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
 
 | Spark NLP | Apache Spark 3.5.x | Apache Spark 3.4.x | Apache Spark 3.3.x | Apache Spark 3.2.x | Apache Spark 3.1.x | Apache Spark 3.0.x | Apache Spark 2.4.x | Apache Spark 2.3.x |
 |-----------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|
+| 6.0.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.5.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.4.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.3.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
@@ -178,6 +179,7 @@ Find out more about `Spark NLP` versions from our [release notes](https://github
 
 | Spark NLP | Python 3.6 | Python 3.7 | Python 3.8 | Python 3.9 | Python 3.10| Scala 2.11 | Scala 2.12 |
 |-----------|------------|------------|------------|------------|------------|------------|------------|
+| 6.0.x     | NO         | YES        | YES        | YES        | YES        | NO         | YES        |
 | 5.5.x     | NO         | YES        | YES        | YES        | YES        | NO         | YES        |
 | 5.4.x     | NO         | YES        | YES        | YES        | YES        | NO         | YES        |
 | 5.3.x     | NO         | YES        | YES        | YES        | YES        | NO         | YES        |
@@ -189,7 +191,7 @@ Find out more about 4.x `SparkNLP` versions in our official [documentation](http
 
 ### Databricks Support
 
-Spark NLP 5.5.3 has been tested and is compatible with the following runtimes:
+Spark NLP 6.0.1 has been tested and is compatible with the following runtimes:
 
 | **CPU**            | **GPU**            |
 |--------------------|--------------------|
@@ -206,7 +208,7 @@ We are compatible with older runtimes. For a full list check databricks support
 
 ### EMR Support
 
-Spark NLP 5.5.3 has been tested and is compatible with the following EMR releases:
+Spark NLP 6.0.1 has been tested and is compatible with the following EMR releases:
 
 | **EMR Release**    |
 |--------------------|
@@ -216,6 +218,13 @@ Spark NLP 5.5.3 has been tested and is compatible with the following EMR release
 | emr-7.0.0          |
 | emr-7.1.0          |
 | emr-7.2.0          |
+| emr-7.3.0          |
+| emr-7.4.0          |
+| emr-7.5.0          |
+| emr-7.6.0          |
+| emr-7.7.0          |
+| emr-7.8.0          |
+
 
 We are compatible with older EMR releases. For a full list check EMR support in our official [documentation](https://sparknlp.org/docs/en/install#emr-support)
 
@@ -237,7 +246,7 @@ deployed to Maven central. To add any of our packages as a dependency in your ap
 from our official documentation.
 
 If you are interested, there is a simple SBT project for Spark NLP to guide you on how to use it in your
-projects [Spark NLP SBT S5.5.3r](https://github.com/maziyarpanahi/spark-nlp-starter)
+projects [Spark NLP Starter](https://github.com/maziyarpanahi/spark-nlp-starter)
 
 ### Python
 
@@ -246,7 +255,7 @@ Check all available installations for Python in our official [documentation](htt
 
 ### Compiled JARs
 
-To compile the jars from source follow [these instructions](https://sparknlp.org/docs/en/compiled#jars) from our official documenation
+To compile the jars from source follow [these instructions](https://sparknlp.org/docs/en/compiled#jars) from our official documentation
 
 ## Platform-Specific Instructions
 
@@ -266,7 +275,7 @@ For detailed instructions on how to use Spark NLP on supported platforms, please
 
 Spark NLP library and all the pre-trained models/pipelines can be used entirely offline with no access to the Internet.
 Please check [these instructions](https://sparknlp.org/docs/en/install#s3-integration) from our official documentation
-to use Spark NLP offline
+to use Spark NLP offline.
 
 ## Advanced Settings
 
@@ -282,7 +291,7 @@ In Spark NLP we can define S3 locations to:
 
 Please check [these instructions](https://sparknlp.org/docs/en/install#s3-integration) from our official documentation.
 
-## Document5.5.3
+## Documentation
 
 ### Examples
 
@@ -315,7 +324,7 @@ the Spark NLP library:
     keywords = {Spark, Natural language processing, Deep learning, Tensorflow, Cluster},
     abstract = {Spark NLP is a Natural Language Processing (NLP) library built on top of Apache Spark ML. It provides simple, performant & accurate NLP annotations for machine learning pipelines that can scale easily in a distributed environment. Spark NLP comes with 1100+ pretrained pipelines and models in more than 192+ languages. It supports nearly all the NLP tasks and modules that can be used seamlessly in a cluster. Downloaded more than 2.7 million times and experiencing 9x growth since January 2020, Spark NLP is used by 54% of healthcare organizations as the world’s most widely used NLP library in the enterprise.}
     }
-}5.5.3
+}
 ```
 
 ## Community support

{spark-nlp-5.5.3 → spark_nlp-6.0.1}/spark_nlp.egg-info/SOURCES.txt

@@ -6,7 +6,6 @@ com/johnsnowlabs/__init__.py
 com/johnsnowlabs/ml/__init__.py
 com/johnsnowlabs/ml/ai/__init__.py
 com/johnsnowlabs/nlp/__init__.py
-spark_nlp.egg-info/.uuid
 spark_nlp.egg-info/PKG-INFO
 spark_nlp.egg-info/SOURCES.txt
 spark_nlp.egg-info/dependency_links.txt
@@ -39,6 +38,7 @@ sparknlp/annotator/audio/hubert_for_ctc.py
 sparknlp/annotator/audio/wav2vec2_for_ctc.py
 sparknlp/annotator/audio/whisper_for_ctc.py
 sparknlp/annotator/classifier_dl/__init__.py
+sparknlp/annotator/classifier_dl/albert_for_multiple_choice.py
 sparknlp/annotator/classifier_dl/albert_for_question_answering.py
 sparknlp/annotator/classifier_dl/albert_for_sequence_classification.py
 sparknlp/annotator/classifier_dl/albert_for_token_classification.py
@@ -62,6 +62,7 @@ sparknlp/annotator/classifier_dl/distil_bert_for_question_answering.py
 sparknlp/annotator/classifier_dl/distil_bert_for_sequence_classification.py
 sparknlp/annotator/classifier_dl/distil_bert_for_token_classification.py
 sparknlp/annotator/classifier_dl/distil_bert_for_zero_shot_classification.py
+sparknlp/annotator/classifier_dl/distilbert_for_multiple_choice.py
 sparknlp/annotator/classifier_dl/longformer_for_question_answering.py
 sparknlp/annotator/classifier_dl/longformer_for_sequence_classification.py
 sparknlp/annotator/classifier_dl/longformer_for_token_classification.py
@@ -69,24 +70,37 @@ sparknlp/annotator/classifier_dl/mpnet_for_question_answering.py
 sparknlp/annotator/classifier_dl/mpnet_for_sequence_classification.py
 sparknlp/annotator/classifier_dl/mpnet_for_token_classification.py
 sparknlp/annotator/classifier_dl/multi_classifier_dl.py
+sparknlp/annotator/classifier_dl/roberta_for_multiple_choice.py
 sparknlp/annotator/classifier_dl/roberta_for_question_answering.py
 sparknlp/annotator/classifier_dl/roberta_for_sequence_classification.py
 sparknlp/annotator/classifier_dl/roberta_for_token_classification.py
 sparknlp/annotator/classifier_dl/roberta_for_zero_shot_classification.py
 sparknlp/annotator/classifier_dl/sentiment_dl.py
 sparknlp/annotator/classifier_dl/tapas_for_question_answering.py
+sparknlp/annotator/classifier_dl/xlm_roberta_for_multiple_choice.py
 sparknlp/annotator/classifier_dl/xlm_roberta_for_question_answering.py
 sparknlp/annotator/classifier_dl/xlm_roberta_for_sequence_classification.py
 sparknlp/annotator/classifier_dl/xlm_roberta_for_token_classification.py
 sparknlp/annotator/classifier_dl/xlm_roberta_for_zero_shot_classification.py
 sparknlp/annotator/classifier_dl/xlnet_for_sequence_classification.py
 sparknlp/annotator/classifier_dl/xlnet_for_token_classification.py
+sparknlp/annotator/cleaners/__init__.py
+sparknlp/annotator/cleaners/cleaner.py
+sparknlp/annotator/cleaners/extractor.py
 sparknlp/annotator/coref/__init__.py
 sparknlp/annotator/coref/spanbert_coref.py
 sparknlp/annotator/cv/__init__.py
 sparknlp/annotator/cv/blip_for_question_answering.py
 sparknlp/annotator/cv/clip_for_zero_shot_classification.py
 sparknlp/annotator/cv/convnext_for_image_classification.py
+sparknlp/annotator/cv/gemma3_for_multimodal.py
+sparknlp/annotator/cv/janus_for_multimodal.py
+sparknlp/annotator/cv/llava_for_multimodal.py
+sparknlp/annotator/cv/mllama_for_multimodal.py
+sparknlp/annotator/cv/paligemma_for_multimodal.py
+sparknlp/annotator/cv/phi3_vision_for_multimodal.py
+sparknlp/annotator/cv/qwen2vl_transformer.py
+sparknlp/annotator/cv/smolvlm_transformer.py
 sparknlp/annotator/cv/swin_for_image_classification.py
 sparknlp/annotator/cv/vision_encoder_decoder_for_image_captioning.py
 sparknlp/annotator/cv/vit_for_image_classification.py
@@ -157,7 +171,9 @@ sparknlp/annotator/sentiment/sentiment_detector.py
 sparknlp/annotator/sentiment/vivekn_sentiment.py
 sparknlp/annotator/seq2seq/__init__.py
 sparknlp/annotator/seq2seq/auto_gguf_model.py
+sparknlp/annotator/seq2seq/auto_gguf_vision_model.py
 sparknlp/annotator/seq2seq/bart_transformer.py
+sparknlp/annotator/seq2seq/cohere_transformer.py
 sparknlp/annotator/seq2seq/cpm_transformer.py
 sparknlp/annotator/seq2seq/gpt2_transformer.py
 sparknlp/annotator/seq2seq/llama2_transformer.py
@@ -166,6 +182,7 @@ sparknlp/annotator/seq2seq/m2m100_transformer.py
 sparknlp/annotator/seq2seq/marian_transformer.py
 sparknlp/annotator/seq2seq/mistral_transformer.py
 sparknlp/annotator/seq2seq/nllb_transformer.py
+sparknlp/annotator/seq2seq/olmo_transformer.py
 sparknlp/annotator/seq2seq/phi2_transformer.py
 sparknlp/annotator/seq2seq/phi3_transformer.py
 sparknlp/annotator/seq2seq/qwen_transformer.py
@@ -225,6 +242,8 @@ sparknlp/pretrained/pretrained_pipeline.py
 sparknlp/pretrained/resource_downloader.py
 sparknlp/pretrained/utils.py
 sparknlp/reader/__init__.py
+sparknlp/reader/enums.py
+sparknlp/reader/pdf_to_text.py
 sparknlp/reader/sparknlp_reader.py
 sparknlp/training/__init__.py
 sparknlp/training/conll.py

{spark-nlp-5.5.3 → spark_nlp-6.0.1}/sparknlp/__init__.py

@@ -132,7 +132,7 @@ def start(gpu=False,
         The initiated Spark session.
 
     """
-    current_version = "5.5.3"
+    current_version = "6.0.1"
 
     if params is None:
         params = {}
@@ -316,4 +316,4 @@ def version():
     str
         The current Spark NLP version.
     """
-    return '5.5.3'
+    return '6.0.1'

{spark-nlp-5.5.3 → spark_nlp-6.0.1}/sparknlp/annotator/classifier_dl/__init__.py

@@ -55,3 +55,7 @@ from sparknlp.annotator.classifier_dl.mpnet_for_token_classification import *
 from sparknlp.annotator.classifier_dl.albert_for_zero_shot_classification import *
 from sparknlp.annotator.classifier_dl.camembert_for_zero_shot_classification import *
 from sparknlp.annotator.classifier_dl.bert_for_multiple_choice import *
+from sparknlp.annotator.classifier_dl.xlm_roberta_for_multiple_choice import *
+from sparknlp.annotator.classifier_dl.roberta_for_multiple_choice import *
+from sparknlp.annotator.classifier_dl.distilbert_for_multiple_choice import *
+from sparknlp.annotator.classifier_dl.albert_for_multiple_choice import *

spark_nlp-6.0.1/sparknlp/annotator/classifier_dl/albert_for_multiple_choice.py

@@ -0,0 +1,161 @@
+#  Copyright 2017-2024 John Snow Labs
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+from sparknlp.common import *
+
+class AlbertForMultipleChoice(AnnotatorModel,
+                              HasCaseSensitiveProperties,
+                              HasBatchedAnnotate,
+                              HasEngine,
+                              HasMaxSentenceLengthLimit):
+    """AlbertForMultipleChoice can load ALBERT Models with a multiple choice classification head on top
+    (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks.
+
+    Pretrained models can be loaded with :meth:`.pretrained` of the companion
+    object:
+
+    >>> spanClassifier = AlbertForMultipleChoice.pretrained() \\
+    ...     .setInputCols(["document_question", "document_context"]) \\
+    ...     .setOutputCol("answer")
+
+    The default model is ``"albert_base_uncased_multiple_choice"``, if no name is
+    provided.
+
+    For available pretrained models please see the `Models Hub
+    <https://sparknlp.org/models?task=Multiple+Choice>`__.
+
+    To see which models are compatible and how to import them see
+    `Import Transformers into Spark NLP 🚀
+    <https://github.com/JohnSnowLabs/spark-nlp/discussions/5669>`_.
+
+    ====================== ======================
+    Input Annotation types Output Annotation type
+    ====================== ======================
+    ``DOCUMENT, DOCUMENT``    ``CHUNK``
+    ====================== ======================
+
+    Parameters
+    ----------
+    batchSize
+        Batch size. Large values allows faster processing but requires more
+        memory, by default 8
+    caseSensitive
+        Whether to ignore case in tokens for embeddings matching, by default
+        False
+    maxSentenceLength
+        Max sentence length to process, by default 512
+
+    Examples
+    --------
+    >>> import sparknlp
+    >>> from sparknlp.base import *
+    >>> from sparknlp.annotator import *
+    >>> from pyspark.ml import Pipeline
+    >>> documentAssembler = MultiDocumentAssembler() \\
+    ...     .setInputCols(["question", "context"]) \\
+    ...     .setOutputCols(["document_question", "document_context"])
+    >>> questionAnswering = AlbertForMultipleChoice.pretrained() \\
+    ...     .setInputCols(["document_question", "document_context"]) \\
+    ...     .setOutputCol("answer") \\
+    ...     .setCaseSensitive(False)
+    >>> pipeline = Pipeline().setStages([
+    ...     documentAssembler,
+    ...     questionAnswering
+    ... ])
+    >>> data = spark.createDataFrame([["The Eiffel Tower is located in which country??", "Germany, France, Italy"]]).toDF("question", "context")
+    >>> result = pipeline.fit(data).transform(data)
+    >>> result.select("answer.result").show(truncate=False)
+    +--------------------+
+    |result              |
+    +--------------------+
+    |[France]             |
+    +--------------------+
+    """
+    name = "AlbertForMultipleChoice"
+
+    inputAnnotatorTypes = [AnnotatorType.DOCUMENT, AnnotatorType.DOCUMENT]
+
+    outputAnnotatorType = AnnotatorType.CHUNK
+
+    choicesDelimiter = Param(Params._dummy(),
+                             "choicesDelimiter",
+                             "Delimiter character use to split the choices",
+                             TypeConverters.toString)
+
+    def setChoicesDelimiter(self, value):
+        """Sets delimiter character use to split the choices
+
+        Parameters
+        ----------
+        value : string
+            Delimiter character use to split the choices
+        """
+        return self._set(caseSensitive=value)
+
+    @keyword_only
+    def __init__(self, classname="com.johnsnowlabs.nlp.annotators.classifier.dl.AlbertForMultipleChoice",
+                 java_model=None):
+        super(AlbertForMultipleChoice, self).__init__(
+            classname=classname,
+            java_model=java_model
+        )
+        self._setDefault(
+            batchSize=4,
+            maxSentenceLength=512,
+            caseSensitive=False,
+            choicesDelimiter = ","
+        )
+
+    @staticmethod
+    def loadSavedModel(folder, spark_session):
+        """Loads a locally saved model.
+
+        Parameters
+        ----------
+        folder : str
+            Folder of the saved model
+        spark_session : pyspark.sql.SparkSession
+            The current SparkSession
+
+        Returns
+        -------
+        BertForQuestionAnswering
+            The restored model
+        """
+        from sparknlp.internal import _AlbertMultipleChoiceLoader
+        jModel = _AlbertMultipleChoiceLoader(folder, spark_session._jsparkSession)._java_obj
+        return AlbertForMultipleChoice(java_model=jModel)
+
+    @staticmethod
+    def pretrained(name="albert_base_uncased_multiple_choice", lang="en", remote_loc=None):
+        """Downloads and loads a pretrained model.
+
+        Parameters
+        ----------
+        name : str, optional
+            Name of the pretrained model, by default
+            "bert_base_uncased_multiple_choice"
+        lang : str, optional
+            Language of the pretrained model, by default "en"
+        remote_loc : str, optional
+            Optional remote address of the resource, by default None. Will use
+            Spark NLPs repositories otherwise.
+
+        Returns
+        -------
+        BertForQuestionAnswering
+            The restored model
+        """
+        from sparknlp.pretrained import ResourceDownloader
+        return ResourceDownloader.downloadModel(AlbertForMultipleChoice, name, lang, remote_loc)

{spark-nlp-5.5.3 → spark_nlp-6.0.1}/sparknlp/annotator/classifier_dl/bert_for_multiple_choice.py

@@ -130,7 +130,7 @@ class BertForMultipleChoice(AnnotatorModel,
 
         Returns
         -------
-        BertForQuestionAnswering
+        BertForMultipleChoice
             The restored model
         """
         from sparknlp.internal import _BertMultipleChoiceLoader
@@ -154,7 +154,7 @@ class BertForMultipleChoice(AnnotatorModel,
 
         Returns
         -------
-        BertForQuestionAnswering
+        BertForMultipleChoice
             The restored model
         """
         from sparknlp.pretrained import ResourceDownloader