spark-nlp 6.0.0__py2.py3-none-any.whl → 6.0.1rc1__py2.py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Potentially problematic release.


This version of spark-nlp might be problematic. Click here for more details.

@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: spark-nlp
3
- Version: 6.0.0
3
+ Version: 6.0.1rc1
4
4
  Summary: John Snow Labs Spark NLP is a natural language processing library built on top of Apache Spark ML. It provides simple, performant & accurate NLP annotations for machine learning pipelines, that scale easily in a distributed environment.
5
5
  Home-page: https://github.com/JohnSnowLabs/spark-nlp
6
6
  Author: John Snow Labs
@@ -95,7 +95,7 @@ $ java -version
95
95
  $ conda create -n sparknlp python=3.7 -y
96
96
  $ conda activate sparknlp
97
97
  # spark-nlp by default is based on pyspark 3.x
98
- $ pip install spark-nlp==6.0.0 pyspark==3.3.1
98
+ $ pip install spark-nlp==6.0.1 pyspark==3.3.1
99
99
  ```
100
100
 
101
101
  In Python console or Jupyter `Python3` kernel:
@@ -161,7 +161,7 @@ For a quick example of using pipelines and models take a look at our official [d
161
161
 
162
162
  ### Apache Spark Support
163
163
 
164
- Spark NLP *6.0.0* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
164
+ Spark NLP *6.0.1* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
165
165
 
166
166
  | Spark NLP | Apache Spark 3.5.x | Apache Spark 3.4.x | Apache Spark 3.3.x | Apache Spark 3.2.x | Apache Spark 3.1.x | Apache Spark 3.0.x | Apache Spark 2.4.x | Apache Spark 2.3.x |
167
167
  |-----------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|
@@ -191,7 +191,7 @@ Find out more about 4.x `SparkNLP` versions in our official [documentation](http
191
191
 
192
192
  ### Databricks Support
193
193
 
194
- Spark NLP 6.0.0 has been tested and is compatible with the following runtimes:
194
+ Spark NLP 6.0.1 has been tested and is compatible with the following runtimes:
195
195
 
196
196
  | **CPU** | **GPU** |
197
197
  |--------------------|--------------------|
@@ -208,7 +208,7 @@ We are compatible with older runtimes. For a full list check databricks support
208
208
 
209
209
  ### EMR Support
210
210
 
211
- Spark NLP 6.0.0 has been tested and is compatible with the following EMR releases:
211
+ Spark NLP 6.0.1 has been tested and is compatible with the following EMR releases:
212
212
 
213
213
  | **EMR Release** |
214
214
  |--------------------|
@@ -246,7 +246,7 @@ deployed to Maven central. To add any of our packages as a dependency in your ap
246
246
  from our official documentation.
247
247
 
248
248
  If you are interested, there is a simple SBT project for Spark NLP to guide you on how to use it in your
249
- projects [Spark NLP SBT S6.0.0r](https://github.com/maziyarpanahi/spark-nlp-starter)
249
+ projects [Spark NLP Starter](https://github.com/maziyarpanahi/spark-nlp-starter)
250
250
 
251
251
  ### Python
252
252
 
@@ -3,7 +3,7 @@ com/johnsnowlabs/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,
3
3
  com/johnsnowlabs/ml/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
4
4
  com/johnsnowlabs/ml/ai/__init__.py,sha256=YQiK2M7U4d8y5irPy_HB8ae0mSpqS9583MH44pnKJXc,295
5
5
  com/johnsnowlabs/nlp/__init__.py,sha256=DPIVXtONO5xXyOk-HB0-sNiHAcco17NN13zPS_6Uw8c,294
6
- sparknlp/__init__.py,sha256=GU7UqzU4QlCwwqWf8vhUopEgQg1RQlyYGnO1hbn4f8U,13783
6
+ sparknlp/__init__.py,sha256=bmd2osfOpjsl6fkGAziSKpz0F5tFXASPI-9L-0GJatQ,13791
7
7
  sparknlp/annotation.py,sha256=I5zOxG5vV2RfPZfqN9enT1i4mo6oBcn3Lrzs37QiOiA,5635
8
8
  sparknlp/annotation_audio.py,sha256=iRV_InSVhgvAwSRe9NTbUH9v6OGvTM-FPCpSAKVu0mE,1917
9
9
  sparknlp/annotation_image.py,sha256=xhCe8Ko-77XqWVuuYHFrjKqF6zPd8Z-RY_rmZXNwCXU,2547
@@ -63,7 +63,7 @@ sparknlp/annotator/classifier_dl/mpnet_for_question_answering.py,sha256=w9hHLrQb
63
63
  sparknlp/annotator/classifier_dl/mpnet_for_sequence_classification.py,sha256=M__giFElL6Q3I88QD6OoXDzdQDk_Zp5sS__Kh_XpLdo,7308
64
64
  sparknlp/annotator/classifier_dl/mpnet_for_token_classification.py,sha256=SgFAJcv7ZE3BmJOehK_CjAaueqaaK6PR33zA5aE9-Ww,6754
65
65
  sparknlp/annotator/classifier_dl/multi_classifier_dl.py,sha256=ylKQzS7ROyeKeiOF4BZiIkQV1sfrnfUUQ9LXFSFK_Vo,16045
66
- sparknlp/annotator/classifier_dl/roberta_for_multiple_choice.py,sha256=A6tB80L1P_xX6MAEbGvCod44-YGBwKep0b-zhdb3PIY,6008
66
+ sparknlp/annotator/classifier_dl/roberta_for_multiple_choice.py,sha256=SlzkA_fKurWOQDhvWlEBiMUfLgIoaRRglIdENMv7u38,6008
67
67
  sparknlp/annotator/classifier_dl/roberta_for_question_answering.py,sha256=WRxu1uhXnY9C4UHdtJ8qiVGhPSX7sCdSaML0AWHOdJw,6471
68
68
  sparknlp/annotator/classifier_dl/roberta_for_sequence_classification.py,sha256=z97uH5WkG8kPX1Y9qtpLwD7egl0kzbVmxtq4xzZgNNI,7857
69
69
  sparknlp/annotator/classifier_dl/roberta_for_token_classification.py,sha256=hvnG31FonfirdLcIy4_bkhbdQalRlqS8x3woScQeRVg,7220
@@ -82,15 +82,18 @@ sparknlp/annotator/cleaners/cleaner.py,sha256=r_0ImrtGT-S-ytOknKoP844FVSv0J9YVKQ
82
82
  sparknlp/annotator/cleaners/extractor.py,sha256=nml8mnOToZYPF5fTp9VWdDfnWTXryLDzp3RWfQoJkWY,5805
83
83
  sparknlp/annotator/coref/__init__.py,sha256=SG8MAaVxQpoYYAsyKaoOlvlHjorDzj3DHt4nnEdBWm8,53
84
84
  sparknlp/annotator/coref/spanbert_coref.py,sha256=AXWJhvVquY2uoApO_Np1fz7_KyJhxnZB4i-xk78sBfc,8407
85
- sparknlp/annotator/cv/__init__.py,sha256=3IavOgDEUnXUK49Kfqya6bjTT8JJWZ6nm-Dqz4lRezY,1298
85
+ sparknlp/annotator/cv/__init__.py,sha256=ySHQ_8pumJFESLUtDqvb0X9oX6He-w_-Jw--Z3ASU5w,1473
86
86
  sparknlp/annotator/cv/blip_for_question_answering.py,sha256=At7L5pPBNDR1r-JGLKM5b3dTrq5Ecz9r0M1gToUVZTs,6551
87
87
  sparknlp/annotator/cv/clip_for_zero_shot_classification.py,sha256=_1pLc9BiFrFN10eJPCDJLJT-vdnTSG9OnB25Y_kKJIA,7528
88
88
  sparknlp/annotator/cv/convnext_for_image_classification.py,sha256=KzaAlYW5M2l73zUozzgg8_p14eGDz9k9PYVAUZLN25k,11874
89
+ sparknlp/annotator/cv/gemma3_for_multimodal.py,sha256=dh0KjTJGqpD-yN7d2f2auMbKLwL5w74Rhgai5y0LeHw,13053
89
90
  sparknlp/annotator/cv/janus_for_multimodal.py,sha256=-TlAfeZ3A8iMJ23Q05Tx_KObgBfy9-qYAN9gAPCvjbw,14499
90
91
  sparknlp/annotator/cv/llava_for_multimodal.py,sha256=kzOcZs08yCnB9AgaogZG28SguGqVUw9sumijhM5YRFU,12064
91
92
  sparknlp/annotator/cv/mllama_for_multimodal.py,sha256=e4_bOGycy-gPYrl8en0mOP3eF8p17Xt85nwE5kmez5g,13071
93
+ sparknlp/annotator/cv/paligemma_for_multimodal.py,sha256=nqwGWRG4kc7FJ26DMwwhkN7FnBWGBZsTjL9H-scs69Q,11204
92
94
  sparknlp/annotator/cv/phi3_vision_for_multimodal.py,sha256=MPGj07Gi-QCE5Ew5l3_SqUqBIR4Tvhhi1ZVbkuX-ihU,12127
93
95
  sparknlp/annotator/cv/qwen2vl_transformer.py,sha256=S2jFwMfh-2iaTl7t8SndH1U1dHSpnlW6E0IQBtw_Xak,12565
96
+ sparknlp/annotator/cv/smolvlm_transformer.py,sha256=JJINJfo_tUvgZ89AG3YqJQ99yjqZpUzJ5vEfISXOeh0,16963
94
97
  sparknlp/annotator/cv/swin_for_image_classification.py,sha256=iZ1KY0GInbQmGzkmuNbds4PGPwCheLXc-Syv2HRmqug,10694
95
98
  sparknlp/annotator/cv/vision_encoder_decoder_for_image_captioning.py,sha256=rEWJte-qN6PI6ML2cGhsZ37wAzjHUtN_WD5pcKAez7M,10167
96
99
  sparknlp/annotator/cv/vit_for_image_classification.py,sha256=D2V3pxAd3rBi1817lxVOqaVvCw4trcVyorQgIPdLNAE,9148
@@ -219,7 +222,7 @@ sparknlp/common/read_as.py,sha256=imxPGwV7jr4Li_acbo0OAHHRGCBbYv-akzEGaBWEfcY,12
219
222
  sparknlp/common/recursive_annotator_approach.py,sha256=vqugBw22cE3Ff7PIpRlnYFuOlchgL0nM26D8j-NdpqU,1449
220
223
  sparknlp/common/storage.py,sha256=D91H3p8EIjNspjqAYu6ephRpCUtdcAir4_PrAbkIQWE,4842
221
224
  sparknlp/common/utils.py,sha256=Yne6yYcwKxhOZC-U4qfYoDhWUP_6BIaAjI5X_P_df1E,1306
222
- sparknlp/internal/__init__.py,sha256=tQSxdp_H1J9IheM176kNnHsSB4fGkQd-39DzCNT82hk,38171
225
+ sparknlp/internal/__init__.py,sha256=YtsUXuuHzv4lATbepu7BhWJEc7Vo65OtEgphxEHOa5Q,39168
223
226
  sparknlp/internal/annotator_java_ml.py,sha256=UGPoThG0rGXUOXGSQnDzEDW81Mu1s5RPF29v7DFyE3c,1187
224
227
  sparknlp/internal/annotator_transformer.py,sha256=fXmc2IWXGybqZpbEU9obmbdBYPc798y42zvSB4tqV9U,1448
225
228
  sparknlp/internal/extended_java_wrapper.py,sha256=hwP0133-hDiDf5sBF-P3MtUsuuDj1PpQbtGZQIRwzfk,2240
@@ -232,7 +235,8 @@ sparknlp/pretrained/pretrained_pipeline.py,sha256=lquxiaABuA68Rmu7csamJPqBoRJqMU
232
235
  sparknlp/pretrained/resource_downloader.py,sha256=8_-rpvO2LsX_Lq4wMPif2ca3RlJZWEabt8pDm2xymiI,7806
233
236
  sparknlp/pretrained/utils.py,sha256=T1MrvW_DaWk_jcOjVLOea0NMFE9w8fe0ZT_5urZ_nEY,1099
234
237
  sparknlp/reader/__init__.py,sha256=-Toj3AIBki-zXPpV8ezFTI2LX1yP_rK2bhpoa8nBkTw,685
235
- sparknlp/reader/pdf_to_text.py,sha256=6AR0rQaIqdcVMSQaM3Vqc9rKQBiQrDf1TxmMUMKfvJw,2208
238
+ sparknlp/reader/enums.py,sha256=MNGug9oJ1BBLM1Pbske13kAabalDzHa2kucF5xzFpHs,770
239
+ sparknlp/reader/pdf_to_text.py,sha256=o2-ZqioR3-apGDo5WCb0_I0sEQr6O-CGMfMb4W4YSss,3892
236
240
  sparknlp/reader/sparknlp_reader.py,sha256=BEKfT9JaOWlA2ddsMNiC-pVRrM9Ad_4J4-Ur3iCNKH0,38218
237
241
  sparknlp/training/__init__.py,sha256=qREi9u-5Vc2VjpL6-XZsyvu5jSEIdIhowW7_kKaqMqo,852
238
242
  sparknlp/training/conll.py,sha256=wKBiSTrjc6mjsl7Nyt6B8f4yXsDJkZb-sn8iOjix9cE,6961
@@ -264,7 +268,7 @@ sparknlp/training/_tf_graph_builders_1x/ner_dl/dataset_encoder.py,sha256=R4yHFN3
264
268
  sparknlp/training/_tf_graph_builders_1x/ner_dl/ner_model.py,sha256=EoCSdcIjqQ3wv13MAuuWrKV8wyVBP0SbOEW41omHlR0,23189
265
269
  sparknlp/training/_tf_graph_builders_1x/ner_dl/ner_model_saver.py,sha256=k5CQ7gKV6HZbZMB8cKLUJuZxoZWlP_DFWdZ--aIDwsc,2356
266
270
  sparknlp/training/_tf_graph_builders_1x/ner_dl/sentence_grouper.py,sha256=pAxjWhjazSX8Vg0MFqJiuRVw1IbnQNSs-8Xp26L4nko,870
267
- spark_nlp-6.0.0.dist-info/METADATA,sha256=sMWEEgYZ_miYyE66W4tWv0QIllVWJ183XT6wSHRB3BM,19581
268
- spark_nlp-6.0.0.dist-info/WHEEL,sha256=bb2Ot9scclHKMOLDEHY6B2sicWOgugjFKaJsT7vwMQo,110
269
- spark_nlp-6.0.0.dist-info/top_level.txt,sha256=uuytur4pyMRw2H_txNY2ZkaucZHUs22QF8-R03ch_-E,13
270
- spark_nlp-6.0.0.dist-info/RECORD,,
271
+ spark_nlp-6.0.1rc1.dist-info/METADATA,sha256=z-li7N02SZLSg5k3RXH9w_GbQKh1ro03sRnl030YIQ8,19580
272
+ spark_nlp-6.0.1rc1.dist-info/WHEEL,sha256=AHX6tWk3qWuce7vKLrj7lnulVHEdWoltgauo8bgCXgU,109
273
+ spark_nlp-6.0.1rc1.dist-info/top_level.txt,sha256=uuytur4pyMRw2H_txNY2ZkaucZHUs22QF8-R03ch_-E,13
274
+ spark_nlp-6.0.1rc1.dist-info/RECORD,,
@@ -1,5 +1,5 @@
1
1
  Wheel-Version: 1.0
2
- Generator: bdist_wheel (0.38.4)
2
+ Generator: setuptools (75.1.0)
3
3
  Root-Is-Purelib: true
4
4
  Tag: py2-none-any
5
5
  Tag: py3-none-any
sparknlp/__init__.py CHANGED
@@ -132,7 +132,7 @@ def start(gpu=False,
132
132
  The initiated Spark session.
133
133
 
134
134
  """
135
- current_version = "6.0.0"
135
+ current_version = "6.0.1-rc1"
136
136
 
137
137
  if params is None:
138
138
  params = {}
@@ -316,4 +316,4 @@ def version():
316
316
  str
317
317
  The current Spark NLP version.
318
318
  """
319
- return '6.0.0'
319
+ return '6.0.1-rc1'
@@ -104,7 +104,7 @@ class RoBertaForMultipleChoice(AnnotatorModel,
104
104
  return self._set(caseSensitive=value)
105
105
 
106
106
  @keyword_only
107
- def __init__(self, classname="com.johnsnowlabs.nlp.annotators.classifier.dl.RobertaForMultipleChoice",
107
+ def __init__(self, classname="com.johnsnowlabs.nlp.annotators.classifier.dl.RoBertaForMultipleChoice",
108
108
  java_model=None):
109
109
  super(RoBertaForMultipleChoice, self).__init__(
110
110
  classname=classname,
@@ -22,3 +22,6 @@ from sparknlp.annotator.cv.mllama_for_multimodal import *
22
22
  from sparknlp.annotator.cv.qwen2vl_transformer import *
23
23
  from sparknlp.annotator.cv.llava_for_multimodal import *
24
24
  from sparknlp.annotator.cv.phi3_vision_for_multimodal import *
25
+ from sparknlp.annotator.cv.smolvlm_transformer import *
26
+ from sparknlp.annotator.cv.paligemma_for_multimodal import *
27
+ from sparknlp.annotator.cv.gemma3_for_multimodal import *
@@ -0,0 +1,351 @@
1
+ # Copyright 2017-2024 John Snow Labs
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
14
+
15
+ from sparknlp.common import *
16
+
17
+ class Gemma3ForMultiModal(AnnotatorModel,
18
+ HasBatchedAnnotateImage,
19
+ HasImageFeatureProperties,
20
+ HasEngine,
21
+ HasGeneratorProperties):
22
+ """Gemma3ForMultiModal can load Gemma 3 Vision models for visual question answering.
23
+ The model consists of a vision encoder, a text encoder, a text decoder and a model merger.
24
+ The vision encoder will encode the input image, the text encoder will encode the input text,
25
+ the model merger will merge the image and text embeddings, and the text decoder will output the answer.
26
+
27
+ Gemma 3 is a family of lightweight, state-of-the-art open models from Google, built from the same
28
+ research and technology used to create the Gemini models. It features:
29
+ - Large 128K context window
30
+ - Multilingual support in over 140 languages
31
+ - Multimodal capabilities handling both text and image inputs
32
+ - Optimized for deployment on limited resources (laptops, desktops, cloud)
33
+
34
+ Pretrained models can be loaded with :meth:`.pretrained` of the companion
35
+ object:
36
+
37
+ >>> visualQA = Gemma3ForMultiModal.pretrained() \
38
+ ... .setInputCols(["image_assembler"]) \
39
+ ... .setOutputCol("answer")
40
+
41
+ The default model is ``"gemma3_4b_it_int4"``, if no name is
42
+ provided.
43
+
44
+ For available pretrained models please see the `Models Hub
45
+ <https://sparknlp.org/models?task=Question+Answering>`__.
46
+
47
+ ====================== ======================
48
+ Input Annotation types Output Annotation type
49
+ ====================== ======================
50
+ ``IMAGE`` ``DOCUMENT``
51
+ ====================== ======================
52
+
53
+ Parameters
54
+ ----------
55
+ batchSize
56
+ Batch size. Large values allows faster processing but requires more
57
+ memory, by default 1
58
+ minOutputLength
59
+ Minimum length of the sequence to be generated, by default 0
60
+ maxOutputLength
61
+ Maximum length of output text, by default 20
62
+ doSample
63
+ Whether or not to use sampling; use greedy decoding otherwise, by default False
64
+ temperature
65
+ The value used to module the next token probabilities, by default 0.6
66
+ topK
67
+ The number of highest probability vocabulary tokens to keep for top-k-filtering, by default -1
68
+ topP
69
+ If set to float < 1, only the most probable tokens with probabilities that add up to ``top_p`` or higher are kept for generation, by default 0.9
70
+ repetitionPenalty
71
+ The parameter for repetition penalty. 1.0 means no penalty, by default 1.0
72
+ noRepeatNgramSize
73
+ If set to int > 0, all ngrams of that size can only occur once, by default 3
74
+ beamSize
75
+ The Number of beams for beam search, by default 1
76
+ maxInputLength
77
+ Maximum length of input text, by default 4096
78
+
79
+ Examples
80
+ --------
81
+ >>> import sparknlp
82
+ >>> from sparknlp.base import *
83
+ >>> from sparknlp.annotator import *
84
+ >>> from pyspark.ml import Pipeline
85
+ >>> from pyspark.sql.functions import lit
86
+ >>>
87
+ >>> imageDF = spark.read.format("image").load(images_path)
88
+ >>> testDF = imageDF.withColumn("text", lit("<bos><start_of_turn>user\nYou are a helpful assistant.\n\n<start_of_image>Describe this image in detail.<end_of_turn>\n<start_of_turn>model\n"))
89
+ >>>
90
+ >>> imageAssembler = ImageAssembler() \
91
+ ... .setInputCol("image") \
92
+ ... .setOutputCol("image_assembler")
93
+ >>>
94
+ >>> visualQA = Gemma3ForMultiModal.pretrained() \
95
+ ... .setInputCols("image_assembler") \
96
+ ... .setOutputCol("answer")
97
+ >>>
98
+ >>> pipeline = Pipeline().setStages([
99
+ ... imageAssembler,
100
+ ... visualQA
101
+ ... ])
102
+ >>>
103
+ >>> result = pipeline.fit(testDF).transform(testDF)
104
+ >>> result.select("image_assembler.origin", "answer.result").show(truncate=False)
105
+ """
106
+
107
+ name = "Gemma3ForMultiModal"
108
+
109
+ inputAnnotatorTypes = [AnnotatorType.IMAGE]
110
+
111
+ outputAnnotatorType = AnnotatorType.DOCUMENT
112
+
113
+ configProtoBytes = Param(Params._dummy(),
114
+ "configProtoBytes",
115
+ "ConfigProto from tensorflow, serialized into byte array. Get with "
116
+ "config_proto.SerializeToString()",
117
+ TypeConverters.toListInt)
118
+
119
+ minOutputLength = Param(Params._dummy(), "minOutputLength", "Minimum length of the sequence to be generated",
120
+ typeConverter=TypeConverters.toInt)
121
+
122
+ maxOutputLength = Param(Params._dummy(), "maxOutputLength", "Maximum length of output text",
123
+ typeConverter=TypeConverters.toInt)
124
+
125
+ doSample = Param(Params._dummy(), "doSample", "Whether or not to use sampling; use greedy decoding otherwise",
126
+ typeConverter=TypeConverters.toBoolean)
127
+
128
+ temperature = Param(Params._dummy(), "temperature", "The value used to module the next token probabilities",
129
+ typeConverter=TypeConverters.toFloat)
130
+
131
+ topK = Param(Params._dummy(), "topK",
132
+ "The number of highest probability vocabulary tokens to keep for top-k-filtering",
133
+ typeConverter=TypeConverters.toInt)
134
+
135
+ topP = Param(Params._dummy(), "topP",
136
+ "If set to float < 1, only the most probable tokens with probabilities that add up to ``top_p`` or higher are kept for generation",
137
+ typeConverter=TypeConverters.toFloat)
138
+
139
+ repetitionPenalty = Param(Params._dummy(), "repetitionPenalty",
140
+ "The parameter for repetition penalty. 1.0 means no penalty. See `this paper <https://arxiv.org/pdf/1909.05858.pdf>`__ for more details",
141
+ typeConverter=TypeConverters.toFloat)
142
+
143
+ noRepeatNgramSize = Param(Params._dummy(), "noRepeatNgramSize",
144
+ "If set to int > 0, all ngrams of that size can only occur once",
145
+ typeConverter=TypeConverters.toInt)
146
+
147
+ ignoreTokenIds = Param(Params._dummy(), "ignoreTokenIds",
148
+ "A list of token ids which are ignored in the decoder's output",
149
+ typeConverter=TypeConverters.toListInt)
150
+ beamSize = Param(Params._dummy(), "beamSize",
151
+ "The Number of beams for beam search.",
152
+ typeConverter=TypeConverters.toInt)
153
+
154
+ maxInputLength = Param(Params._dummy(), "maxInputLength", "Maximum length of input text",
155
+ typeConverter=TypeConverters.toInt)
156
+
157
+ def setMaxSentenceSize(self, value):
158
+ """Sets Maximum sentence length that the annotator will process, by
159
+ default 50.
160
+
161
+ Parameters
162
+ ----------
163
+ value : int
164
+ Maximum sentence length that the annotator will process
165
+ """
166
+ return self._set(maxSentenceLength=value)
167
+
168
+ def setIgnoreTokenIds(self, value):
169
+ """A list of token ids which are ignored in the decoder's output.
170
+
171
+ Parameters
172
+ ----------
173
+ value : List[int]
174
+ The words to be filtered out
175
+ """
176
+ return self._set(ignoreTokenIds=value)
177
+
178
+ def setConfigProtoBytes(self, b):
179
+ """Sets configProto from tensorflow, serialized into byte array.
180
+
181
+ Parameters
182
+ ----------
183
+ b : List[int]
184
+ ConfigProto from tensorflow, serialized into byte array
185
+ """
186
+ return self._set(configProtoBytes=b)
187
+
188
+ def setMinOutputLength(self, value):
189
+ """Sets minimum length of the sequence to be generated.
190
+
191
+ Parameters
192
+ ----------
193
+ value : int
194
+ Minimum length of the sequence to be generated
195
+ """
196
+ return self._set(minOutputLength=value)
197
+
198
+ def setMaxOutputLength(self, value):
199
+ """Sets maximum length of output text.
200
+
201
+ Parameters
202
+ ----------
203
+ value : int
204
+ Maximum length of output text
205
+ """
206
+ return self._set(maxOutputLength=value)
207
+
208
+ def setDoSample(self, value):
209
+ """Sets whether or not to use sampling, use greedy decoding otherwise.
210
+
211
+ Parameters
212
+ ----------
213
+ value : bool
214
+ Whether or not to use sampling; use greedy decoding otherwise
215
+ """
216
+ return self._set(doSample=value)
217
+
218
+ def setTemperature(self, value):
219
+ """Sets the value used to module the next token probabilities.
220
+
221
+ Parameters
222
+ ----------
223
+ value : float
224
+ The value used to module the next token probabilities
225
+ """
226
+ return self._set(temperature=value)
227
+
228
+ def setTopK(self, value):
229
+ """Sets the number of highest probability vocabulary tokens to keep for
230
+ top-k-filtering.
231
+
232
+ Parameters
233
+ ----------
234
+ value : int
235
+ Number of highest probability vocabulary tokens to keep
236
+ """
237
+ return self._set(topK=value)
238
+
239
+ def setTopP(self, value):
240
+ """Sets the top cumulative probability for vocabulary tokens.
241
+
242
+ If set to float < 1, only the most probable tokens with probabilities
243
+ that add up to ``topP`` or higher are kept for generation.
244
+
245
+ Parameters
246
+ ----------
247
+ value : float
248
+ Cumulative probability for vocabulary tokens
249
+ """
250
+ return self._set(topP=value)
251
+
252
+ def setRepetitionPenalty(self, value):
253
+ """Sets the parameter for repetition penalty. 1.0 means no penalty.
254
+
255
+ Parameters
256
+ ----------
257
+ value : float
258
+ The repetition penalty
259
+
260
+ References
261
+ ----------
262
+ See `Ctrl: A Conditional Transformer Language Model For Controllable
263
+ Generation <https://arxiv.org/pdf/1909.05858.pdf>`__ for more details.
264
+ """
265
+ return self._set(repetitionPenalty=value)
266
+
267
+ def setNoRepeatNgramSize(self, value):
268
+ """Sets size of n-grams that can only occur once.
269
+
270
+ If set to int > 0, all ngrams of that size can only occur once.
271
+
272
+ Parameters
273
+ ----------
274
+ value : int
275
+ N-gram size can only occur once
276
+ """
277
+ return self._set(noRepeatNgramSize=value)
278
+
279
+ def setBeamSize(self, value):
280
+ """Sets the number of beam size for beam search, by default `4`.
281
+
282
+ Parameters
283
+ ----------
284
+ value : int
285
+ Number of beam size for beam search
286
+ """
287
+ return self._set(beamSize=value)
288
+
289
+ @keyword_only
290
+ def __init__(self, classname="com.johnsnowlabs.nlp.annotators.cv.Gemma3ForMultiModal",
291
+ java_model=None):
292
+ super(Gemma3ForMultiModal, self).__init__(
293
+ classname=classname,
294
+ java_model=java_model
295
+ )
296
+ self._setDefault(
297
+ batchSize=1,
298
+ minOutputLength=0,
299
+ maxOutputLength=20,
300
+ doSample=False,
301
+ temperature=0.6,
302
+ topK=-1,
303
+ topP=0.9,
304
+ repetitionPenalty=1.0,
305
+ noRepeatNgramSize=3,
306
+ ignoreTokenIds=[],
307
+ beamSize=1,
308
+ maxInputLength=4096,
309
+ )
310
+
311
+ @staticmethod
312
+ def loadSavedModel(folder, spark_session, use_openvino=False):
313
+ """Loads a locally saved model.
314
+
315
+ Parameters
316
+ ----------
317
+ folder : str
318
+ Folder of the saved model
319
+ spark_session : pyspark.sql.SparkSession
320
+ The current SparkSession
321
+
322
+ Returns
323
+ -------
324
+ Gemma3ForMultiModal
325
+ The restored model
326
+ """
327
+ from sparknlp.internal import _Gemma3ForMultiModalLoader
328
+ jModel = _Gemma3ForMultiModalLoader(folder, spark_session._jsparkSession, use_openvino)._java_obj
329
+ return Gemma3ForMultiModal(java_model=jModel)
330
+
331
+ @staticmethod
332
+ def pretrained(name="gemma3_4b_it_int4", lang="en", remote_loc=None):
333
+ """Downloads and loads a pretrained model.
334
+
335
+ Parameters
336
+ ----------
337
+ name : str, optional
338
+ Name of the pretrained model, by default "gemma3_4b_it_int4"
339
+ lang : str, optional
340
+ Language of the pretrained model, by default "en"
341
+ remote_loc : str, optional
342
+ Optional remote address of the resource, by default None. Will use
343
+ Spark NLPs repositories otherwise.
344
+
345
+ Returns
346
+ -------
347
+ Gemma3ForMultiModal
348
+ The restored model
349
+ """
350
+ from sparknlp.pretrained import ResourceDownloader
351
+ return ResourceDownloader.downloadModel(Gemma3ForMultiModal, name, lang, remote_loc)