spark-nlp 5.2.3__py2.py3-none-any.whl → 5.3.0__py2.py3-none-any.whl

{spark_nlp-5.2.3.dist-info → spark_nlp-5.3.0.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: spark-nlp
-Version: 5.2.3
+Version: 5.3.0
 Summary: John Snow Labs Spark NLP is a natural language processing library built on top of Apache Spark ML. It provides simple, performant & accurate NLP annotations for machine learning pipelines, that scale easily in a distributed environment.
 Home-page: https://github.com/JohnSnowLabs/spark-nlp
 Author: John Snow Labs
@@ -54,7 +54,7 @@ environment.
 Spark NLP comes with **36000+** pretrained **pipelines** and **models** in more than **200+** languages.
 It also offers tasks such as **Tokenization**, **Word Segmentation**, **Part-of-Speech Tagging**, Word and Sentence **Embeddings**, **Named Entity Recognition**, **Dependency Parsing**, **Spell Checking**, **Text Classification**, **Sentiment Analysis**, **Token Classification**, **Machine Translation** (+180 languages), **Summarization**, **Question Answering**, **Table Question Answering**, **Text Generation**, **Image Classification**, **Image to Text (captioning)**, **Automatic Speech Recognition**, **Zero-Shot Learning**, and many more [NLP tasks](#features).
 
-**Spark NLP** is the only open-source NLP library in **production** that offers state-of-the-art transformers such as **BERT**, **CamemBERT**, **ALBERT**, **ELECTRA**, **XLNet**, **DistilBERT**, **RoBERTa**, **DeBERTa**, **XLM-RoBERTa**, **Longformer**, **ELMO**, **Universal Sentence Encoder**, **Facebook BART**, **Instructor**, **E5**, **Google T5**, **MarianMT**, **OpenAI GPT2**, **Vision Transformers (ViT)**, **OpenAI Whisper**, and many more not only to **Python** and **R**, but also to **JVM** ecosystem (**Java**, **Scala**, and **Kotlin**) at **scale** by extending **Apache Spark** natively.
+**Spark NLP** is the only open-source NLP library in **production** that offers state-of-the-art transformers such as **BERT**, **CamemBERT**, **ALBERT**, **ELECTRA**, **XLNet**, **DistilBERT**, **RoBERTa**, **DeBERTa**, **XLM-RoBERTa**, **Longformer**, **ELMO**, **Universal Sentence Encoder**, **Llama-2**, **M2M100**, **BART**, **Instructor**, **E5**, **Google T5**, **MarianMT**, **OpenAI GPT2**, **Vision Transformers (ViT)**, **OpenAI Whisper**, and many more not only to **Python** and **R**, but also to **JVM** ecosystem (**Java**, **Scala**, and **Kotlin**) at **scale** by extending **Apache Spark** natively.
 
 ## Project's website
 
@@ -143,42 +143,34 @@ documentation and examples
 - BERT Sentence Embeddings (TF Hub & HuggingFace models)
 - RoBerta Sentence Embeddings (HuggingFace models)
 - XLM-RoBerta Sentence Embeddings (HuggingFace models)
-- Instructor Embeddings (HuggingFace models)
+- INSTRUCTOR Embeddings (HuggingFace models)
 - E5 Embeddings (HuggingFace models)
 - MPNet Embeddings (HuggingFace models)
 - OpenAI Embeddings
-- Sentence Embeddings
-- Chunk Embeddings
+- Sentence & Chunk Embeddings
 - Unsupervised keywords extraction
 - Language Detection & Identification (up to 375 languages)
-- Multi-class Sentiment analysis (Deep learning)
-- Multi-label Sentiment analysis (Deep learning)
+- Multi-class & Multi-labe Sentiment analysis (Deep learning)
 - Multi-class Text Classification (Deep learning)
-- BERT for Token & Sequence Classification
-- DistilBERT for Token & Sequence Classification
-- CamemBERT for Token & Sequence Classification
-- ALBERT for Token & Sequence Classification
-- RoBERTa for Token & Sequence Classification
-- DeBERTa for Token & Sequence Classification
-- XLM-RoBERTa for Token & Sequence Classification
+- BERT for Token & Sequence Classification & Question Answering
+- DistilBERT for Token & Sequence Classification & Question Answering
+- CamemBERT for Token & Sequence Classification & Question Answering
+- ALBERT for Token & Sequence Classification & Question Answering
+- RoBERTa for Token & Sequence Classification & Question Answering
+- DeBERTa for Token & Sequence Classification & Question Answering
+- XLM-RoBERTa for Token & Sequence Classification & Question Answering
+- Longformer for Token & Sequence Classification & Question Answering
+- MPnet for Token & Sequence Classification & Question Answering
 - XLNet for Token & Sequence Classification
-- Longformer for Token & Sequence Classification
-- BERT for Token & Sequence Classification
-- BERT for Question Answering
-- CamemBERT for Question Answering
-- DistilBERT for Question Answering
-- ALBERT for Question Answering
-- RoBERTa for Question Answering
-- DeBERTa for Question Answering
-- XLM-RoBERTa for Question Answering
-- Longformer for Question Answering
-- Table Question Answering (TAPAS)
 - Zero-Shot NER Model
 - Zero-Shot Text Classification by Transformers (ZSL)
 - Neural Machine Translation (MarianMT)
+- Many-to-Many multilingual translation model (Facebook M2M100)
+- Table Question Answering (TAPAS)
 - Text-To-Text Transfer Transformer (Google T5)
 - Generative Pre-trained Transformer 2 (OpenAI GPT2)
 - Seq2Seq for NLG, Translation, and Comprehension (Facebook BART)
+- Chat and Conversational LLMs (Facebook Llama-22)
 - Vision Transformer (Google ViT)
 - Swin Image Classification (Microsoft Swin Transformer)
 - ConvNext Image Classification (Facebook ConvNext)
@@ -205,7 +197,7 @@ To use Spark NLP you need the following requirements:
 
 **GPU (optional):**
 
-Spark NLP 5.2.3 is built with ONNX 1.16.3 and TensorFlow 2.7.1 deep learning engines. The minimum following NVIDIA® software are only required for GPU support:
+Spark NLP 5.3.0 is built with ONNX 1.17.0 and TensorFlow 2.7.1 deep learning engines. The minimum following NVIDIA® software are only required for GPU support:
 
 - NVIDIA® GPU drivers version 450.80.02 or higher
 - CUDA® Toolkit 11.2
@@ -221,7 +213,7 @@ $ java -version
 $ conda create -n sparknlp python=3.7 -y
 $ conda activate sparknlp
 # spark-nlp by default is based on pyspark 3.x
-$ pip install spark-nlp==5.2.3 pyspark==3.3.1
+$ pip install spark-nlp==5.3.0 pyspark==3.3.1
 ```
 
 In Python console or Jupyter `Python3` kernel:
@@ -266,11 +258,12 @@ For more examples, you can visit our dedicated [examples](https://github.com/Joh
 
 ## Apache Spark Support
 
-Spark NLP *5.2.3* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
+Spark NLP *5.3.0* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
 
 | Spark NLP | Apache Spark 3.5.x | Apache Spark 3.4.x | Apache Spark 3.3.x | Apache Spark 3.2.x | Apache Spark 3.1.x | Apache Spark 3.0.x | Apache Spark 2.4.x | Apache Spark 2.3.x |
 |-----------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|
-| 5.2.x     | YES          | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
+| 5.3.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
+| 5.2.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.1.x     | Partially          | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 5.0.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
 | 4.4.x     | YES                | YES                | YES                | YES                | YES                | YES                | NO                 | NO                 |
@@ -291,6 +284,7 @@ Find out more about `Spark NLP` versions from our [release notes](https://github
 
 | Spark NLP | Python 3.6 | Python 3.7 | Python 3.8 | Python 3.9 | Python 3.10| Scala 2.11 | Scala 2.12 |
 |-----------|------------|------------|------------|------------|------------|------------|------------|
+| 5.3.x     | NO         | YES        | YES        | YES        | YES        | NO         | YES        |
 | 5.2.x     | NO         | YES        | YES        | YES        | YES        | NO         | YES        |
 | 5.1.x     | NO         | YES        | YES        | YES        | YES        | NO         | YES        |
 | 5.0.x     | NO         | YES        | YES        | YES        | YES        | NO         | YES        |
@@ -308,7 +302,7 @@ Find out more about `Spark NLP` versions from our [release notes](https://github
 
 ## Databricks Support
 
-Spark NLP 5.2.3 has been tested and is compatible with the following runtimes:
+Spark NLP 5.3.0 has been tested and is compatible with the following runtimes:
 
 **CPU:**
 
@@ -350,6 +344,10 @@ Spark NLP 5.2.3 has been tested and is compatible with the following runtimes:
 - 14.0 ML
 - 14.1
 - 14.1 ML
+- 14.2
+- 14.2 ML
+- 14.3
+- 14.3 ML
 
 **GPU:**
 
@@ -372,10 +370,12 @@ Spark NLP 5.2.3 has been tested and is compatible with the following runtimes:
 - 13.3 ML & GPU
 - 14.0 ML & GPU
 - 14.1 ML & GPU
+- 14.2 ML & GPU
+- 14.3 ML & GPU
 
 ## EMR Support
 
-Spark NLP 5.2.3 has been tested and is compatible with the following EMR releases:
+Spark NLP 5.3.0 has been tested and is compatible with the following EMR releases:
 
 - emr-6.2.0
 - emr-6.3.0
@@ -391,8 +391,11 @@ Spark NLP 5.2.3 has been tested and is compatible with the following EMR release
 - emr-6.12.0
 - emr-6.13.0
 - emr-6.14.0
+- emr-6.15.0
+- emr-7.0.0
 
 Full list of [Amazon EMR 6.x releases](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-6x.html)
+Full list of [Amazon EMR 7.x releases](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-7x.html)
 
 NOTE: The EMR 6.1.0 and 6.1.1 are not supported.
 
@@ -422,11 +425,11 @@ Spark NLP supports all major releases of Apache Spark 3.0.x, Apache Spark 3.1.x,
 ```sh
 # CPU
 
-spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.2.3
+spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.3.0
 
-pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.2.3
+pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.3.0
 
-spark-submit --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.2.3
+spark-submit --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.3.0
 ```
 
 The `spark-nlp` has been published to
@@ -435,11 +438,11 @@ the [Maven Repository](https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/s
 ```sh
 # GPU
 
-spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.2.3
+spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.3.0
 
-pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.2.3
+pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.3.0
 
-spark-submit --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.2.3
+spark-submit --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.3.0
 
 ```
 
@@ -449,11 +452,11 @@ the [Maven Repository](https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/s
 ```sh
 # AArch64
 
-spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.2.3
+spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.3.0
 
-pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.2.3
+pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.3.0
 
-spark-submit --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.2.3
+spark-submit --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.3.0
 
 ```
 
@@ -463,11 +466,11 @@ the [Maven Repository](https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/s
 ```sh
 # M1/M2 (Apple Silicon)
 
-spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.2.3
+spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.3.0
 
-pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.2.3
+pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.3.0
 
-spark-submit --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.2.3
+spark-submit --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.3.0
 
 ```
 
@@ -481,7 +484,7 @@ set in your SparkSession:
 spark-shell \
   --driver-memory 16g \
   --conf spark.kryoserializer.buffer.max=2000M \
-  --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.2.3
+  --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.3.0
 ```
 
 ## Scala
@@ -499,7 +502,7 @@ coordinates:
 <dependency>
     <groupId>com.johnsnowlabs.nlp</groupId>
     <artifactId>spark-nlp_2.12</artifactId>
-    <version>5.2.3</version>
+    <version>5.3.0</version>
 </dependency>
 ```
 
@@ -510,7 +513,7 @@ coordinates:
 <dependency>
     <groupId>com.johnsnowlabs.nlp</groupId>
     <artifactId>spark-nlp-gpu_2.12</artifactId>
-    <version>5.2.3</version>
+    <version>5.3.0</version>
 </dependency>
 ```
 
@@ -521,7 +524,7 @@ coordinates:
 <dependency>
     <groupId>com.johnsnowlabs.nlp</groupId>
     <artifactId>spark-nlp-aarch64_2.12</artifactId>
-    <version>5.2.3</version>
+    <version>5.3.0</version>
 </dependency>
 ```
 
@@ -532,7 +535,7 @@ coordinates:
 <dependency>
     <groupId>com.johnsnowlabs.nlp</groupId>
     <artifactId>spark-nlp-silicon_2.12</artifactId>
-    <version>5.2.3</version>
+    <version>5.3.0</version>
 </dependency>
 ```
 
@@ -542,28 +545,28 @@ coordinates:
 
 ```sbtshell
 // https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp
-libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp" % "5.2.3"
+libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp" % "5.3.0"
 ```
 
 **spark-nlp-gpu:**
 
 ```sbtshell
 // https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-gpu
-libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp-gpu" % "5.2.3"
+libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp-gpu" % "5.3.0"
 ```
 
 **spark-nlp-aarch64:**
 
 ```sbtshell
 // https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-aarch64
-libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp-aarch64" % "5.2.3"
+libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp-aarch64" % "5.3.0"
 ```
 
 **spark-nlp-silicon:**
 
 ```sbtshell
 // https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-silicon
-libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp-silicon" % "5.2.3"
+libraryDependencies += "com.johnsnowlabs.nlp" %% "spark-nlp-silicon" % "5.3.0"
 ```
 
 Maven
@@ -585,7 +588,7 @@ If you installed pyspark through pip/conda, you can install `spark-nlp` through
 Pip:
 
 ```bash
-pip install spark-nlp==5.2.3
+pip install spark-nlp==5.3.0
 ```
 
 Conda:
@@ -614,7 +617,7 @@ spark = SparkSession.builder
     .config("spark.driver.memory", "16G")
     .config("spark.driver.maxResultSize", "0")
     .config("spark.kryoserializer.buffer.max", "2000M")
-    .config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.12:5.2.3")
+    .config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.12:5.3.0")
     .getOrCreate()
 ```
 
@@ -685,7 +688,7 @@ Use either one of the following options
 - Add the following Maven Coordinates to the interpreter's library list
 
 ```bash
-com.johnsnowlabs.nlp:spark-nlp_2.12:5.2.3
+com.johnsnowlabs.nlp:spark-nlp_2.12:5.3.0
 ```
 
 - Add a path to pre-built jar from [here](#compiled-jars) in the interpreter's library list making sure the jar is
@@ -696,7 +699,7 @@ com.johnsnowlabs.nlp:spark-nlp_2.12:5.2.3
 Apart from the previous step, install the python module through pip
 
 ```bash
-pip install spark-nlp==5.2.3
+pip install spark-nlp==5.3.0
 ```
 
 Or you can install `spark-nlp` from inside Zeppelin by using Conda:
@@ -724,7 +727,7 @@ launch the Jupyter from the same Python environment:
 $ conda create -n sparknlp python=3.8 -y
 $ conda activate sparknlp
 # spark-nlp by default is based on pyspark 3.x
-$ pip install spark-nlp==5.2.3 pyspark==3.3.1 jupyter
+$ pip install spark-nlp==5.3.0 pyspark==3.3.1 jupyter
 $ jupyter notebook
 ```
 
@@ -741,7 +744,7 @@ export PYSPARK_PYTHON=python3
 export PYSPARK_DRIVER_PYTHON=jupyter
 export PYSPARK_DRIVER_PYTHON_OPTS=notebook
 
-pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.2.3
+pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.3.0
 ```
 
 Alternatively, you can mix in using `--jars` option for pyspark + `pip install spark-nlp`
@@ -768,7 +771,7 @@ This script comes with the two options to define `pyspark` and `spark-nlp` versi
 # -s is for spark-nlp
 # -g will enable upgrading libcudnn8 to 8.1.0 on Google Colab for GPU usage
 # by default they are set to the latest
-!wget https://setup.johnsnowlabs.com/colab.sh -O - | bash /dev/stdin -p 3.2.3 -s 5.2.3
+!wget https://setup.johnsnowlabs.com/colab.sh -O - | bash /dev/stdin -p 3.2.3 -s 5.3.0
 ```
 
 [Spark NLP quick start on Google Colab](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp/blob/master/examples/python/quick_start_google_colab.ipynb)
@@ -791,7 +794,7 @@ This script comes with the two options to define `pyspark` and `spark-nlp` versi
 # -s is for spark-nlp
 # -g will enable upgrading libcudnn8 to 8.1.0 on Kaggle for GPU usage
 # by default they are set to the latest
-!wget https://setup.johnsnowlabs.com/colab.sh -O - | bash /dev/stdin -p 3.2.3 -s 5.2.3
+!wget https://setup.johnsnowlabs.com/colab.sh -O - | bash /dev/stdin -p 3.2.3 -s 5.3.0
 ```
 
 [Spark NLP quick start on Kaggle Kernel](https://www.kaggle.com/mozzie/spark-nlp-named-entity-recognition) is a live
@@ -810,9 +813,9 @@ demo on Kaggle Kernel that performs named entity recognitions by using Spark NLP
 
 3. In `Libraries` tab inside your cluster you need to follow these steps:
 
-   3.1. Install New -> PyPI -> `spark-nlp==5.2.3` -> Install
+   3.1. Install New -> PyPI -> `spark-nlp==5.3.0` -> Install
 
-   3.2. Install New -> Maven -> Coordinates -> `com.johnsnowlabs.nlp:spark-nlp_2.12:5.2.3` -> Install
+   3.2. Install New -> Maven -> Coordinates -> `com.johnsnowlabs.nlp:spark-nlp_2.12:5.3.0` -> Install
 
 4. Now you can attach your notebook to the cluster and use Spark NLP!
 
@@ -863,7 +866,7 @@ A sample of your software configuration in JSON on S3 (must be public access):
       "spark.kryoserializer.buffer.max": "2000M",
       "spark.serializer": "org.apache.spark.serializer.KryoSerializer",
       "spark.driver.maxResultSize": "0",
-      "spark.jars.packages": "com.johnsnowlabs.nlp:spark-nlp_2.12:5.2.3"
+      "spark.jars.packages": "com.johnsnowlabs.nlp:spark-nlp_2.12:5.3.0"
     }
 }]
 ```
@@ -872,7 +875,7 @@ A sample of AWS CLI to launch EMR cluster:
 
 ```.sh
 aws emr create-cluster \
---name "Spark NLP 5.2.3" \
+--name "Spark NLP 5.3.0" \
 --release-label emr-6.2.0 \
 --applications Name=Hadoop Name=Spark Name=Hive \
 --instance-type m4.4xlarge \
@@ -936,7 +939,7 @@ gcloud dataproc clusters create ${CLUSTER_NAME} \
   --enable-component-gateway \
   --metadata 'PIP_PACKAGES=spark-nlp spark-nlp-display google-cloud-bigquery google-cloud-storage' \
   --initialization-actions gs://goog-dataproc-initialization-actions-${REGION}/python/pip-install.sh \
-  --properties spark:spark.serializer=org.apache.spark.serializer.KryoSerializer,spark:spark.driver.maxResultSize=0,spark:spark.kryoserializer.buffer.max=2000M,spark:spark.jars.packages=com.johnsnowlabs.nlp:spark-nlp_2.12:5.2.3
+  --properties spark:spark.serializer=org.apache.spark.serializer.KryoSerializer,spark:spark.driver.maxResultSize=0,spark:spark.kryoserializer.buffer.max=2000M,spark:spark.jars.packages=com.johnsnowlabs.nlp:spark-nlp_2.12:5.3.0
 ```
 
 2. On an existing one, you need to install spark-nlp and spark-nlp-display packages from PyPI.
@@ -979,7 +982,7 @@ spark = SparkSession.builder
     .config("spark.kryoserializer.buffer.max", "2000m")
     .config("spark.jsl.settings.pretrained.cache_folder", "sample_data/pretrained")
     .config("spark.jsl.settings.storage.cluster_tmp_dir", "sample_data/storage")
-    .config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.12:5.2.3")
+    .config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.12:5.3.0")
     .getOrCreate()
 ```
 
@@ -993,7 +996,7 @@ spark-shell \
   --conf spark.kryoserializer.buffer.max=2000M \
   --conf spark.jsl.settings.pretrained.cache_folder="sample_data/pretrained" \
   --conf spark.jsl.settings.storage.cluster_tmp_dir="sample_data/storage" \
-  --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.2.3
+  --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.3.0
 ```
 
 **pyspark:**
@@ -1006,7 +1009,7 @@ pyspark \
   --conf spark.kryoserializer.buffer.max=2000M \
   --conf spark.jsl.settings.pretrained.cache_folder="sample_data/pretrained" \
   --conf spark.jsl.settings.storage.cluster_tmp_dir="sample_data/storage" \
-  --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.2.3
+  --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.3.0
 ```
 
 **Databricks:**
@@ -1278,7 +1281,7 @@ spark = SparkSession.builder
     .config("spark.driver.memory", "16G")
     .config("spark.driver.maxResultSize", "0")
     .config("spark.kryoserializer.buffer.max", "2000M")
-    .config("spark.jars", "/tmp/spark-nlp-assembly-5.2.3.jar")
+    .config("spark.jars", "/tmp/spark-nlp-assembly-5.3.0.jar")
     .getOrCreate()
 ```
 
@@ -1287,7 +1290,7 @@ spark = SparkSession.builder
   version (3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x)
 - If you are local, you can load the Fat JAR from your local FileSystem, however, if you are in a cluster setup you need
   to put the Fat JAR on a distributed FileSystem such as HDFS, DBFS, S3, etc. (
-  i.e., `hdfs:///tmp/spark-nlp-assembly-5.2.3.jar`)
+  i.e., `hdfs:///tmp/spark-nlp-assembly-5.3.0.jar`)
 
 Example of using pretrained Models and Pipelines in offline:
 

{spark_nlp-5.2.3.dist-info → spark_nlp-5.3.0.dist-info}/RECORD

@@ -1,7 +1,7 @@
 com/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
 com/johnsnowlabs/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
 com/johnsnowlabs/nlp/__init__.py,sha256=DPIVXtONO5xXyOk-HB0-sNiHAcco17NN13zPS_6Uw8c,294
-sparknlp/__init__.py,sha256=qzDxFYDRyF2Jw1kVlbunQjoL6qtiJ5EA9td1vsm1J5w,13588
+sparknlp/__init__.py,sha256=Q51KCuoFbaMIgFWGe-we8xtE3uhZuEC3wFc2XeZZcMU,13588
 sparknlp/annotation.py,sha256=I5zOxG5vV2RfPZfqN9enT1i4mo6oBcn3Lrzs37QiOiA,5635
 sparknlp/annotation_audio.py,sha256=iRV_InSVhgvAwSRe9NTbUH9v6OGvTM-FPCpSAKVu0mE,1917
 sparknlp/annotation_image.py,sha256=xhCe8Ko-77XqWVuuYHFrjKqF6zPd8Z-RY_rmZXNwCXU,2547
@@ -22,13 +22,13 @@ sparknlp/annotator/n_gram_generator.py,sha256=KRX5xfxmorOfYQkQHZWkkXjwjC13gDTAXn
 sparknlp/annotator/normalizer.py,sha256=7AkAOB-e8b2uyUBwYoq9HvMPijOwV3wEoxcB3BVsr4w,8780
 sparknlp/annotator/stemmer.py,sha256=Tl48voyG9wqbT5MAA1hDKW90NorU8rIDhttJxOo1s3Q,2948
 sparknlp/annotator/stop_words_cleaner.py,sha256=Z9yI9AWDIAXbPM2X6n84voiW31Z20XofCL-tTQNo5ro,7015
-sparknlp/annotator/tf_ner_dl_graph_builder.py,sha256=jYciU0fTWg4q1MSwPG5oha6vMeALIDdCK8TPR8-78hg,6373
+sparknlp/annotator/tf_ner_dl_graph_builder.py,sha256=ovsRBUfw9lJkuetmrcYRmW1Ll-33sdDPi4xJ0M_Fs7k,6379
 sparknlp/annotator/token2_chunk.py,sha256=FtS2Doav9xL1IrC9ZUU4iXqyipp-iT3g68kZt-7YCcQ,2674
 sparknlp/annotator/audio/__init__.py,sha256=dXjtvi5c0aTZFq1Q_JciUd1uFTBVSJoUdcq0hiYd8yk,757
 sparknlp/annotator/audio/hubert_for_ctc.py,sha256=76PfwPZZvOHU5kfDqLueCFbmqa4W8pMNRGoCvOqjsEA,7859
 sparknlp/annotator/audio/wav2vec2_for_ctc.py,sha256=K78P1U6vA4O1UufsLYzy0H7arsKNmwPcIV7kzDFsA5Q,6210
 sparknlp/annotator/audio/whisper_for_ctc.py,sha256=uII51umuohqwnAW0Q7VdxEFyr_j5LMnfpcRlf8TbetA,9800
-sparknlp/annotator/classifier_dl/__init__.py,sha256=WYVdQaqdVEbOm3guZsoiNAPDPkli7qt3fx3xhUU2stQ,3444
+sparknlp/annotator/classifier_dl/__init__.py,sha256=tGg78A8LUgobZFre_3ySN51KGNyl0Zx0inxT9yfL_g8,3686
 sparknlp/annotator/classifier_dl/albert_for_question_answering.py,sha256=LG2dL6Fky1T35yXTUZBfIihIIGnkRFQ7ECQ3HRXXEG8,6517
 sparknlp/annotator/classifier_dl/albert_for_sequence_classification.py,sha256=kWx7f9pcKE2qw319gn8FN0Md5dX38gbmfeoY9gWCLNk,7842
 sparknlp/annotator/classifier_dl/albert_for_token_classification.py,sha256=5rdsjWnsAVmtP-idU7ATKJ8lkH2rtlKZLnpi4Mq27eI,6839
@@ -36,7 +36,7 @@ sparknlp/annotator/classifier_dl/bart_for_zero_shot_classification.py,sha256=yqQ
 sparknlp/annotator/classifier_dl/bert_for_question_answering.py,sha256=2euY_RAdMPA4IHJXZAd5MkQojFOtFNhB_hSc1iVQ5DQ,6433
 sparknlp/annotator/classifier_dl/bert_for_sequence_classification.py,sha256=AzD3RQcRuQc0DDTbL6vGiacTtHlZnbAqksNvRQq7EQE,7800
 sparknlp/annotator/classifier_dl/bert_for_token_classification.py,sha256=uJXoDLPfPWiRmKqtw_3lLBvneIirj87S2JWwfd33zq8,6668
-sparknlp/annotator/classifier_dl/bert_for_zero_shot_classification.py,sha256=0DW8bn257A8NO4uOss2meBDnbCcASDT7cQHQzbvT-X0,8351
+sparknlp/annotator/classifier_dl/bert_for_zero_shot_classification.py,sha256=mli7_TZjrFs6GPwWtgpPty6HrRKIXrEZKjcR00NKyBo,8318
 sparknlp/annotator/classifier_dl/camembert_for_question_answering.py,sha256=BeE-62tFkXMoyiy3PtcnwgT2-wqzTFo5VZHrWUqsWmM,6510
 sparknlp/annotator/classifier_dl/camembert_for_sequence_classification.py,sha256=06bkwhNBcmNS5gR_JrMjBDW3jAdjEI5YL4SuV16Va7E,7962
 sparknlp/annotator/classifier_dl/camembert_for_token_classification.py,sha256=vjwDE_kZiBupENaYvUZOTTqVOb3KCsGse-QX3QOutz4,6522
@@ -44,6 +44,7 @@ sparknlp/annotator/classifier_dl/classifier_dl.py,sha256=Dj-T5ByCgzgFpah7LVz_07Q
 sparknlp/annotator/classifier_dl/deberta_for_question_answering.py,sha256=oikVBeVohsSR9HPV_yq_0U7zHps94UO4lXbYu9G7MF0,6486
 sparknlp/annotator/classifier_dl/deberta_for_sequence_classification.py,sha256=H2LDT8ttD9hxfFDrymsyCq0EwCuWl5FE2-XVqT9LcRQ,7773
 sparknlp/annotator/classifier_dl/deberta_for_token_classification.py,sha256=jj5hB9AV-0Of505E6z62lYPIWmsqNeTX0vRRq3_7T9I,6807
+sparknlp/annotator/classifier_dl/deberta_for_zero_shot_classification.py,sha256=AmCSFpR0xsxwus6spCpiw6zduGtvg4B_lLS5PUDXjvc,8711
 sparknlp/annotator/classifier_dl/distil_bert_for_question_answering.py,sha256=yA4LrI4RN4f44wbIrdpwqderTJBhAkjAHpUxcCeCROE,6552
 sparknlp/annotator/classifier_dl/distil_bert_for_sequence_classification.py,sha256=Cax3LcVLppiHs1dyahsBSq_TLHSwI2-K7LGCZHZNs1I,7926
 sparknlp/annotator/classifier_dl/distil_bert_for_token_classification.py,sha256=y9S83LW0Mfn4fRzopRXFj8l2gb-Nrm1rr9zRftOckJU,6832
@@ -51,6 +52,8 @@ sparknlp/annotator/classifier_dl/distil_bert_for_zero_shot_classification.py,sha
 sparknlp/annotator/classifier_dl/longformer_for_question_answering.py,sha256=VKbOKSTtwdeSsSzB2oKiRlFwSOcpHuMfkvgGM3ofBIo,6553
 sparknlp/annotator/classifier_dl/longformer_for_sequence_classification.py,sha256=_XO3Ufl_wHyUgUIechZ6J1VCE2G2W-FUPZfHmJSfQvk,7932
 sparknlp/annotator/classifier_dl/longformer_for_token_classification.py,sha256=RmiFuBRhIAoJoQ8Rgcu997-PxBK1hhWuLVlS1qztMyk,6848
+sparknlp/annotator/classifier_dl/mpnet_for_question_answering.py,sha256=w9hHLrQbDIUHAdCKiXNDneAbohMKopixAKU2wkYkqbs,5522
+sparknlp/annotator/classifier_dl/mpnet_for_sequence_classification.py,sha256=M__giFElL6Q3I88QD6OoXDzdQDk_Zp5sS__Kh_XpLdo,7308
 sparknlp/annotator/classifier_dl/multi_classifier_dl.py,sha256=ylKQzS7ROyeKeiOF4BZiIkQV1sfrnfUUQ9LXFSFK_Vo,16045
 sparknlp/annotator/classifier_dl/roberta_for_question_answering.py,sha256=WRxu1uhXnY9C4UHdtJ8qiVGhPSX7sCdSaML0AWHOdJw,6471
 sparknlp/annotator/classifier_dl/roberta_for_sequence_classification.py,sha256=z97uH5WkG8kPX1Y9qtpLwD7egl0kzbVmxtq4xzZgNNI,7857
@@ -100,7 +103,7 @@ sparknlp/annotator/embeddings/xlm_roberta_embeddings.py,sha256=t-Bg1bQcqI_fIqUWQ
 sparknlp/annotator/embeddings/xlm_roberta_sentence_embeddings.py,sha256=ojxD3H2VgDEn-RzDdCz0X485pojHBAFrlzsNemI05bY,8602
 sparknlp/annotator/embeddings/xlnet_embeddings.py,sha256=hJrlsJeO3D7uz54xiEiqqXEbq24YGuWz8U652PV9fNE,9336
 sparknlp/annotator/er/__init__.py,sha256=eF9Z-PanVfZWSVN2HSFbE7QjCDb6NYV5ESn6geYKlek,692
-sparknlp/annotator/er/entity_ruler.py,sha256=NFJgUMh6PV6XdzAdONX9icDbGxdBLze9NrOm_lhezPo,8785
+sparknlp/annotator/er/entity_ruler.py,sha256=7eZtAwoixkl88jTyKEqTKf9Wzo459VXQkYmFBozUY6A,8784
 sparknlp/annotator/keyword_extraction/__init__.py,sha256=KotCR238x7LgisinsRGaARgPygWUIwC624FmH-sHacE,720
 sparknlp/annotator/keyword_extraction/yake_keyword_extraction.py,sha256=oeB-8qdMoljG-mgFOCsfnpxyK5jFBZnX7jAUQwsnHTc,13215
 sparknlp/annotator/ld_dl/__init__.py,sha256=gWNGOaozABT83J4Mn7JmNQsXzm27s3PHpMQmlXl-5L8,704
@@ -132,13 +135,15 @@ sparknlp/annotator/sentence/sentence_detector_dl.py,sha256=-Osj9Bm9KyZRTAWkOsK9c
 sparknlp/annotator/sentiment/__init__.py,sha256=Lq3vKaZS1YATLMg0VNXSVtkWL5q5G9taGBvdrvSwnfg,766
 sparknlp/annotator/sentiment/sentiment_detector.py,sha256=m545NGU0Xzg_PO6_qIfpli1uZj7JQcyFgqe9R6wAPFI,8154
 sparknlp/annotator/sentiment/vivekn_sentiment.py,sha256=4rpXWDgzU6ddnbrSCp9VdLb2epCc9oZ3c6XcqxEw8nk,9655
-sparknlp/annotator/seq2seq/__init__.py,sha256=qSfofkO7oQAdgxFct7U-8eTfewEL2V-I_u-EnWnD89s,891
+sparknlp/annotator/seq2seq/__init__.py,sha256=UQK-_3wLkUdW1piGudCx1_k3Tg3tERZJYOBnfMRj8pA,1011
 sparknlp/annotator/seq2seq/bart_transformer.py,sha256=I1flM4yeCzEAKOdQllBC30XuedxVJ7ferkFhZ6gwEbE,18481
 sparknlp/annotator/seq2seq/gpt2_transformer.py,sha256=Oz95R_NRR4tWHu_bW6Ak2832ZILXycp3ify7LfRSi8o,15310
+sparknlp/annotator/seq2seq/llama2_transformer.py,sha256=YPge5f4qfv7XZY_LoH2HRzvbZ--XoTTY_BupxxYaCd8,13862
+sparknlp/annotator/seq2seq/m2m100_transformer.py,sha256=fTFGFWaFfJt5kaLvnYknf_23PVyjBuha48asFsE_NaE,16082
 sparknlp/annotator/seq2seq/marian_transformer.py,sha256=mQ4Ylh7ZzXAOue8f-x0gqzfS3vAz3XUdD7eQ2XhcEs4,13781
 sparknlp/annotator/seq2seq/t5_transformer.py,sha256=wDVxNLluIU1HGZFqaKKc4YTt4l-elPlAtQ7EEa0f5tg,17308
 sparknlp/annotator/similarity/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-sparknlp/annotator/similarity/document_similarity_ranker.py,sha256=2UQwHQA05lkQ9e_Z_YVAbpWFdBv0ac4oJDroqCJNbF0,14226
+sparknlp/annotator/similarity/document_similarity_ranker.py,sha256=OFAXEBuALFJglwThsGK8YaJ_pgW1tcevB7jVq-8SyKM,14991
 sparknlp/annotator/spell_check/__init__.py,sha256=sdnPR3f3Q9mHiv-n4g_O7KpRWPRPweyATSF6Tth_Niw,830
 sparknlp/annotator/spell_check/context_spell_checker.py,sha256=OtjN51K3TyQpFmZrhPrvxZwCJsENFwTkeNKQYWrP-Gw,31992
 sparknlp/annotator/spell_check/norvig_sweeting.py,sha256=6ET9KnAqXIQDJ5U9px1ixUbC6R63ln_ljruvh_oLiwA,13197
@@ -155,13 +160,13 @@ sparknlp/base/audio_assembler.py,sha256=HKa9mXvmuMUrjTihUZkppGj-WJjcUrm2BGapNuPi
 sparknlp/base/doc2_chunk.py,sha256=TyvbdJNkVo9favHlOEoH5JwKbjpk5ZVJ75p8Cilp9jM,6551
 sparknlp/base/document_assembler.py,sha256=zl-SXWMTR3B0EZ8z6SWYchCwEo-61FhU6u7dHUKDIOg,6697
 sparknlp/base/embeddings_finisher.py,sha256=5QU1Okgl2ULrPVf4ze1H0SsRCMYXWGARtUsT7dagBYA,7659
-sparknlp/base/finisher.py,sha256=kQxR50A82xNrP2ainEAXoVWz0ZZG_ZHmcgzBIAwZqos,8577
+sparknlp/base/finisher.py,sha256=V4wkMm9Ug09q4zTQc9T9Wr-awmu2Hu-eNaJ039YgZXM,8583
 sparknlp/base/graph_finisher.py,sha256=a8fxk3ei2YQw6s0Y9Yy8oMOF1i1XUrgqaiwVE0VPt4w,4834
 sparknlp/base/has_recursive_fit.py,sha256=P55rSHLIXhihXWS2bOC_DskcQTc3njieVD1JkjS2bcA,849
 sparknlp/base/has_recursive_transform.py,sha256=UkGNgo4LMsjQC-Coeefg4bJcg7FoPcPiG382zEa6Ywk,841
 sparknlp/base/image_assembler.py,sha256=HytRoYJTLMqGtvScHoFnp6CasG9IVNYAHYiT2_rrmeE,3719
-sparknlp/base/light_pipeline.py,sha256=7uS9RV2dJF6Xjo0qrhdZabNrd82ERMy9SQpTmboU-RY,16541
-sparknlp/base/multi_document_assembler.py,sha256=tos6BYCtfTAplcmP2zphqdqPW9eQbHkzy9t-fwQ33Ww,7064
+sparknlp/base/light_pipeline.py,sha256=Jk2DLpT4PLHCANlOo_WetTdPba_5lYs3ywiyY3lM-PE,16577
+sparknlp/base/multi_document_assembler.py,sha256=4htET1fRAeOB6zhsNXsBq5rKZvn-LGD4vrFRjPZeqow,7070
 sparknlp/base/recursive_pipeline.py,sha256=V9rTnu8KMwgjoceykN9pF1mKGtOkkuiC_n9v8dE3LDk,4279
 sparknlp/base/table_assembler.py,sha256=Kxu3R2fY6JgCxEc07ibsMsjip6dgcPDHLiWAZ8gC_d8,5102
 sparknlp/base/token_assembler.py,sha256=qiHry07L7mVCqeHSH6hHxLygv1AsfZIE4jy1L75L3Do,5075
@@ -177,7 +182,7 @@ sparknlp/common/read_as.py,sha256=imxPGwV7jr4Li_acbo0OAHHRGCBbYv-akzEGaBWEfcY,12
 sparknlp/common/recursive_annotator_approach.py,sha256=vqugBw22cE3Ff7PIpRlnYFuOlchgL0nM26D8j-NdpqU,1449
 sparknlp/common/storage.py,sha256=D91H3p8EIjNspjqAYu6ephRpCUtdcAir4_PrAbkIQWE,4842
 sparknlp/common/utils.py,sha256=Yne6yYcwKxhOZC-U4qfYoDhWUP_6BIaAjI5X_P_df1E,1306
-sparknlp/internal/__init__.py,sha256=Al75CEdkKLyJSwXMzMIETKYo9vczWXtkPTagZ4TGydw,25119
+sparknlp/internal/__init__.py,sha256=g4REY_0X2Sr05szDb9681oiPqRWlT4KaOpcAOj3q32A,26496
 sparknlp/internal/annotator_java_ml.py,sha256=UGPoThG0rGXUOXGSQnDzEDW81Mu1s5RPF29v7DFyE3c,1187
 sparknlp/internal/annotator_transformer.py,sha256=fXmc2IWXGybqZpbEU9obmbdBYPc798y42zvSB4tqV9U,1448
 sparknlp/internal/extended_java_wrapper.py,sha256=hwP0133-hDiDf5sBF-P3MtUsuuDj1PpQbtGZQIRwzfk,2240
@@ -219,7 +224,7 @@ sparknlp/training/_tf_graph_builders_1x/ner_dl/dataset_encoder.py,sha256=R4yHFN3
 sparknlp/training/_tf_graph_builders_1x/ner_dl/ner_model.py,sha256=EoCSdcIjqQ3wv13MAuuWrKV8wyVBP0SbOEW41omHlR0,23189
 sparknlp/training/_tf_graph_builders_1x/ner_dl/ner_model_saver.py,sha256=k5CQ7gKV6HZbZMB8cKLUJuZxoZWlP_DFWdZ--aIDwsc,2356
 sparknlp/training/_tf_graph_builders_1x/ner_dl/sentence_grouper.py,sha256=pAxjWhjazSX8Vg0MFqJiuRVw1IbnQNSs-8Xp26L4nko,870
-spark_nlp-5.2.3.dist-info/METADATA,sha256=QXMxdjxt8d8HEmdpys1UOmdWUvb1KfIdwYhfQ8pnSU0,56589
-spark_nlp-5.2.3.dist-info/WHEEL,sha256=bb2Ot9scclHKMOLDEHY6B2sicWOgugjFKaJsT7vwMQo,110
-spark_nlp-5.2.3.dist-info/top_level.txt,sha256=uuytur4pyMRw2H_txNY2ZkaucZHUs22QF8-R03ch_-E,13
-spark_nlp-5.2.3.dist-info/RECORD,,
+spark_nlp-5.3.0.dist-info/METADATA,sha256=t_H-q3uAb32zOrySfga2iCrqy3oCXZyfws1a7JaCGz8,57087
+spark_nlp-5.3.0.dist-info/WHEEL,sha256=bb2Ot9scclHKMOLDEHY6B2sicWOgugjFKaJsT7vwMQo,110
+spark_nlp-5.3.0.dist-info/top_level.txt,sha256=uuytur4pyMRw2H_txNY2ZkaucZHUs22QF8-R03ch_-E,13
+spark_nlp-5.3.0.dist-info/RECORD,,

sparknlp/__init__.py

@@ -128,7 +128,7 @@ def start(gpu=False,
         The initiated Spark session.
 
     """
-    current_version = "5.2.3"
+    current_version = "5.3.0"
 
     if params is None:
         params = {}
@@ -309,4 +309,4 @@ def version():
     str
         The current Spark NLP version.
     """
-    return '5.2.3'
+    return '5.3.0'

sparknlp/annotator/classifier_dl/__init__.py

@@ -47,4 +47,7 @@ from sparknlp.annotator.classifier_dl.bert_for_zero_shot_classification import *
 from sparknlp.annotator.classifier_dl.distil_bert_for_zero_shot_classification import *
 from sparknlp.annotator.classifier_dl.roberta_for_zero_shot_classification import *
 from sparknlp.annotator.classifier_dl.xlm_roberta_for_zero_shot_classification import *
-from sparknlp.annotator.classifier_dl.bart_for_zero_shot_classification import *
+from sparknlp.annotator.classifier_dl.bart_for_zero_shot_classification import *
+from sparknlp.annotator.classifier_dl.deberta_for_zero_shot_classification import *
+from sparknlp.annotator.classifier_dl.mpnet_for_sequence_classification import *
+from sparknlp.annotator.classifier_dl.mpnet_for_question_answering import *

sparknlp/annotator/classifier_dl/bert_for_zero_shot_classification.py

@@ -41,7 +41,7 @@ class BertForZeroShotClassification(AnnotatorModel,
     ...     .setInputCols(["token", "document"]) \\
     ...     .setOutputCol("label")
 
-    The default model is ``"bert_base_cased_zero_shot_classifier_xnli"``, if no name is
+    The default model is ``"bert_zero_shot_classifier_mnli"``, if no name is
     provided.
 
     For available pretrained models please see the `Models Hub
@@ -189,14 +189,14 @@ class BertForZeroShotClassification(AnnotatorModel,
         return BertForZeroShotClassification(java_model=jModel)
 
     @staticmethod
-    def pretrained(name="bert_base_cased_zero_shot_classifier_xnli", lang="en", remote_loc=None):
+    def pretrained(name="bert_zero_shot_classifier_mnli", lang="xx", remote_loc=None):
         """Downloads and loads a pretrained model.
 
         Parameters
         ----------
         name : str, optional
             Name of the pretrained model, by default
-            "bert_base_cased_zero_shot_classifier_xnli"
+            "bert_zero_shot_classifier_mnli"
         lang : str, optional
             Language of the pretrained model, by default "en"
         remote_loc : str, optional