clarifai 10.0.0__py3-none-any.whl → 10.1.0__py3-none-any.whl

clarifai/models/model_serving/docs/inference_parameters.md

@@ -1,134 +1,139 @@
 ## Inference paramaters
 
-When making prediction, you may need to change some paramaters to adjust the result. Those paramaters will be passed through `paramaters()` of a request in triton python model.
+In order to send it to `inference_parameters` of `predict` in `inference.py`, you can define some parameters and they will be visible and adjustable on Clarifai model view.
 
-In order to send it to `**kwargs` of `get_predictions` in `inference.py`, you can define some parameters and they will be visible and adjustable on Clarifai model view.
+This document helps you to understand the concept of inference parameters and how to add it `clarifai_config.yaml`
 
-This document helps you to create your inference parameters that can be visibale and adjustable easily on Clarifai platform. The defined parameters will be sent as `json` file when you use `clarifai-upload-model` cli.
+## Overview
+
+Each paramter has 4 fields:
 
-### JSON file structure:
-The file contains a list of object has 4 fields:
 * `path` (str): name of your parameter, it must be valid as python variable
 * `field_type` (int): the parameter data type is one of {1,2,21,3}, it means {boolean, string, encrypted_string, number} respectively. `Number` means `int` or `float`. "Encrypted_string is a string that can be used to store your secrets, like API key. The API will not return the values for this as plaintext.
 * `default_value`: a default value of the parameter.
 * `description` (str): short sentence describes what the parameter does
 
-An example of 4 parameters:
-```json
-[
-  {
-    "path": "boolean_var",
-    "field_type": 1,
-    "default_value": true,
-    "description": "a boolean variable"
-  },
-  {
-    "path": "string_var",
-    "field_type": 2,
-    "default_value": "string_1",
-    "description": "a string variable"
-  },
-  {
-    "path": "number_var",
-    "field_type": 3,
-    "default_value": 9.9,
-    "description": "a float number variable"
-  },
-  {
-    "path": "secret_string_var",
-    "field_type": 21,
-    "default_value": "API_KEY",
-    "description": "a string variable contains secret like API key"
-  },
-]
+An example of 4 type parameters:
+
+```yaml
+- path: boolean_var
+  default_value: true
+  field_type: 1
+  description: a boolean variable
+- path: string_var
+  default_value: "a string"
+  field_type: 2
+  description: a string variable
+- path: number_var
+  default_value: 1
+  field_type: 3
+  description: a number variable
+- path: secret_string_var
+  default_value: "YOUR_SECRET"
+  field_type: 21
+  description: a string variable contains secret like API key
 ```
 
-### Generate JSON file
-1. Manually create the file based on above structure
-2. By code:
-
-#### 2.1. Fully setup
-```python
-from clarifai.models.model_serving.model_config.inference_parameter import InferParamManager, InferParam, InferParamType
-
-params = [
-  InferParam(
-    path="boolean_var",
-    field_type=InferParamType.BOOL,
-    default_value=True,
-    description="a boolean varaiabe"
-  ),
-  InferParam(
-    path="string_var",
-    field_type=InferParamType.STRING,
-    default_value="string_1",
-    description="a string varaiabe"
-  ),
-  InferParam(
-    path="number_var",
-    field_type=InferParamType.NUMBER,
-    default_value=9.9,
-    description="a float number varaiabe"
-  ),
-  InferParam(
-    path=secret_string_var",
-    field_type=InferParamType.ENCRYPTED_STRING,
-    default_value="API_KEY",
-    description="a string variable contains secret like API key"
-  ),
-]
-
-ipm = InferParamManager(params=params)
-ipm.export("your_file.json")
+## Add them to the config file
+
+For example with 4 sample paramaters above.
+
+1. Manually:
+Insert them to field inference_parameters of the file, e.g.
+
+```yaml
+clarifai_model:
+  clarifai_model_id: ''
+  clarifai_user_app_id: ''
+  description: ''
+  inference_parameters:
+    - path: boolean_var
+      default_value: true
+      field_type: 1
+      description: a boolean variable
+    - path: string_var
+      default_value: "a string"
+      field_type: 2
+      description: a string variable
+    - path: number_var
+      default_value: 1
+      field_type: 3
+      description: a number variable
+    - path: secret_string_var
+      default_value: "YOUR_SECRET"
+      field_type: 21
+      description: a string variable contains secret like API key
+  labels: []
+  type: text-to-image
+serving_backend:
+  triton:
+    ...
 ```
 
-##### 2.2. Shorten
-`NOTE`: in this way `description` field will be set as empty aka "".
-*You need to modify* `description` in order to be able to upload the settings to Clarifai.
-
-`NOTE`: in this way `ENCRYPTED_STRING` type must be defined with "_" prefix
-
-```python
-params = dict(boolean_var=True, string_var="string_1", number_var=9.9, _secret_string_var="YOUR_KEY")
-ipm = InferParamManager.from_kwargs(**params)
-ipm.export("your_file.json")
-
-```
+2. Semi: If you have a large number of fields, adding them one by one with specific field types can be exhaustive and unsafe.
 
-3. In `test.py`. You can define your paramaters like `2.2. Shorten` in `inference_parameters` attribute of `CustomTestInferenceModel`, the file will be generated when you run the test. Keep in mind to change `description`
+To address this, you can define them as a dictionary, where the key is the path and the value is the default value. Then, inject them into `BaseTest()` in `test.py` within your model repository. For example, suppose your test.py looks like this:
 
-### Usage
-Your defined parameters will be passed through `kwargs` of `InferenceModel.get_predictions` method
-in `inference.py`
 ```python
-class InferenceModel:
-  def __init__():
-    # initialization
-    self.model = YourModel()
-
-  @some_wrapper_function
-  def get_predictions(self, input_data, **kwargs):
-    # `kwargs` contains your inference parameters
+class CustomTest(unittest.TestCase):
 
-    # get a value from kwargs
-    number_var = kwargs.get("number_var", 9.9)
+  def setUp(self) -> None:
+    your_infer_parameter = dict()
+    self.model = BaseTest(your_infer_parameter)
 
-    # pass everything to a function
-    output = self.model.predict(input_data, **kwargs)
-
-    return SomeOutputType(output)
+  def test_default_cases(self):
+    self.model.test_with_default_inputs()
 
 ```
 
-in `test.py`
+The `BaseTest` class takes inference parameters as a dict, then validating their values and finally save to the config file
+With current samples, the file will turn to
+
 ```python
-class CustomTestInferenceModel:
-  inference_parameters = "" # input a path of json file from `2.1` or a dict from `2.2`
+class CustomTest(unittest.TestCase):
+
+  def setUp(self) -> None:
+    your_infer_parameter = dict(boolean_var=True, string_var="a string", number_var=1, float_number_var=0.1, _secret_string_var="YOUR_SECRET")
+    self.model = BaseTest(your_infer_parameter)
 
   ...
+```
 
-  def test_something(self):
-    input = ...
-    output = self.triton_get_predictions(input, number_var=1, string_var="test", _secret="KEY")
-    self.assert(...)
+After run `test.py` with pytest. The config file looks like:
+
+```yaml
+clarifai_model:
+  clarifai_model_id: ''
+  clarifai_user_app_id: ''
+  description: ''
+  inference_parameters:
+    - path: boolean_var
+      default_value: true
+      field_type: 1
+      description: boolean_var
+    - path: string_var
+      default_value: "a string"
+      field_type: 2
+      description: string_var
+    - path: number_var
+      default_value: 1
+      field_type: 3
+      description: number_var
+    - path: float_number_var
+      default_value: 0.1
+      field_type: 3
+      description: float_number_var
+    - path: _secret_string_var
+      default_value: "YOUR_SECRET"
+      field_type: 21
+      description: _secret_string_var
+  labels: []
+  type: text-to-image
+serving_backend:
+  triton:
+    ...
 ```
+
+> [!Note]
+> * `description` field is set as `path`
+> * For `ENCRYPTED_STRING`, it must be defined with `"_" prefix`

clarifai/models/model_serving/docs/model_types.md

@@ -1,20 +1,19 @@
-## Clarifai Model Types
+Each model type requires different input and output types. The table below illustrates the relationship between supported models and their corresponding input and output types.
 
-Models on the clarifai platform are deployed using the [Triton Inference Server Python Backend](https://github.com/triton-inference-server/python_backend) to allow for pre and post processing of data to and from the model.
+| Type                | Input       | Output               |
+|---------------------|-------------|----------------------|
+| multimodal-embedder |  image,text | EmbeddingOutput      |
+| text-classifier     |  text       | ClassifierOutput     |
+| text-embedder       |  text       | EmbeddingOutput      |
+| text-to-image       |  text       | ImageOutput          |
+| text-to-text        |  text       | TextOutput           |
+| visual-classifier   |  image      | ClassifierOutput     |
+| visual-detector     |  image      | VisualDetectorOutput |
+| visual-embedder     |  image      | EmbeddingOutput      |
+| visual-segmenter    |  image      | MasksOutput          |
 
-Inputs into the models are passed as numpy arrays and the predictions are similarly returned as numpy arrays.
-The predictions from user defined models in the [inference script](../README.md#the-inference-script) file have to match certain formats and shapes for the models to be upload compatible.
+Note:
 
-Clarifai [model types](../models/model_types.py) are decorator functions that are responsible for passing input batch requests to user defined inference models to get predictions and format the resultant predictions into Triton Inference responses that are sent by the server for each client inference request.
-
-## Supported Model Types Wrapper Functions:
-
-- visual_detector
-- visual_classifier
-- text_classifier
-- text_to_text
-- text_embedder
-- text_to_image
-- visual_embedder
-- visual_segmenter
-- multimodal_embedder
+* `image`: single image is RGB np.ndarray with shape of [W, H, 3]
+* `text`: single text is a string in python
+* `multimodal`: has more than one input types

clarifai/models/model_serving/model_config/__init__.py

@@ -10,5 +10,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-from .config import *  # noqa # pylint: disable=unused-import
-from .serializer import Serializer  # noqa # pylint: disable=unused-import
+from .base import *  # noqa
+from .config import *  # noqa
+from .inference_parameter import InferParam, InferParamManager  # noqa
+from .output import *  # noqa

clarifai/models/model_serving/model_config/base.py

@@ -0,0 +1,369 @@
+from typing import Dict, Iterable, List, TypedDict, Union
+
+import numpy as np
+
+from ..constants import IMAGE_TENSOR_NAME, TEXT_TENSOR_NAME
+from .config import ModelConfigClass, ModelTypes, get_model_config
+from .output import (ClassifierOutput, EmbeddingOutput, ImageOutput, MasksOutput, TextOutput,
+                     VisualDetectorOutput)
+from .triton import wrappers as triton_wrapper
+
+
+class _TypeCheckModelOutput(type):
+
+  def __new__(cls, name, bases, attrs):
+    """
+    Override child `predict` function with parent._output_type_check(child.predict).
+    Aim to check if child.predict returns valid output type
+    """
+
+    def wrap_function(fn_name, base, base_fn, other_fn):
+
+      def new_fn(_self,
+                 input_data,
+                 inference_parameters: Dict[str, Union[bool, str, float, int]] = {}):
+        # Run child class
+        out = other_fn(_self, input_data, inference_parameters=inference_parameters)
+        # Run type check
+        return base_fn(base, input_data, out)
+
+      new_fn.__name__ = "wrapped_%s" % fn_name
+      new_fn.__doc__ = other_fn.__doc__
+      return new_fn
+
+    if name != "_BaseClarifaiModel":
+      attrs["predict"] = wrap_function("predict", bases[0],
+                                       getattr(bases[0], "_output_type_check", lambda: None),
+                                       attrs.setdefault("predict", lambda: None))
+
+    return type.__new__(cls, name, bases, attrs)
+
+
+class _BaseClarifaiModel(metaclass=_TypeCheckModelOutput):
+  _config: ModelConfigClass = None
+
+  @property
+  def config(self):
+    return self._config
+
+  def _output_type_check(self, input, output):
+    output_type = self._config.clarifai_model.output_type
+    if isinstance(output, Iterable):
+      assert all(
+          each.__class__.__name__ == output_type for each in output
+      ), f"Expected output is iteration of `{output_type}` type, got iteration `{output}`"
+      assert len(output) == len(
+          input
+      ), f"Input length and output length must be equal, but got input length of {len(input)} and output length of {len(output)}"
+    else:
+      raise ValueError(f"Expected output is iteration of `{output_type}` type, got `{output}`")
+    return output
+
+  def predict(self,
+              input_data: Union[List[np.ndarray], Dict[str, List[np.ndarray]]],
+              inference_parameters: Dict[str, Union[bool, str, float, int]] = {}) -> Iterable:
+    """
+    Prediction method.
+
+    Args:
+    -----
+    - input_data: A list of input data item to predict on. The type depends on model input type:
+      * `image`: List[np.ndarray]
+      * `text`: List[str]
+      * `multimodal`:
+        input_data is list of dict where key is input type name e.i. `image`, `text` and value is list.
+        {"image": List[np.ndarray], "text": List[str]}
+
+    - inference_parameters (Dict[str, Union[bool, str, float, int]]): your inference parameters.
+
+    Returns:
+    --------
+      List of one of the `clarifai.models.model_serving.model_config.output` types. Refer to the README/docs
+    """
+    raise NotImplementedError
+
+  def _tritonserver_predict(self,
+                            input_data,
+                            inference_parameters: Dict[str, Union[bool, str, float, int]] = {}):
+    """ This method is invoked within tritonserver, specifically in the model.py of the Python backend. Attempting to execute it outside of the triton environment will result in failure."""
+    raise NotImplementedError
+
+
+_MultiModalInputTypeDict = TypedDict("_MultiModalInputTypeDict", {
+    IMAGE_TENSOR_NAME: np.ndarray,
+    TEXT_TENSOR_NAME: str
+})
+
+
+class MultiModalEmbedder(_BaseClarifaiModel):
+  _config: ModelConfigClass = get_model_config(ModelTypes.multimodal_embedder)
+
+  def predict(self,
+              input_data: List[_MultiModalInputTypeDict],
+              inference_parameters: Dict[str, Union[bool, str, float, int]] = {}
+             ) -> Iterable[EmbeddingOutput]:
+    """ Custom prediction function for `multimodal-embedder` model.
+
+    Args:
+      input_data (List[_MultiModalInputTypeDict]): List of dict of key-value: `image`(np.ndarray) and `text` (str)
+      inference_parameters (Dict[str, Union[bool, str, float, int]]): your inference parameters
+
+    Returns:
+      list of EmbeddingOutput
+    """
+    raise NotImplementedError
+
+  @triton_wrapper.multimodal_embedder
+  def _tritonserver_predict(self,
+                            input_data,
+                            inference_parameters: Dict[str, Union[bool, str, float, int]] = {}):
+    """ This method is invoked within tritonserver, specifically in the model.py of the Python backend. Attempting to execute it outside of the triton environment will result in failure."""
+    return self.predict(input_data, inference_parameters=inference_parameters)
+
+
+class TextClassifier(_BaseClarifaiModel):
+  _config: ModelConfigClass = get_model_config(ModelTypes.text_classifier)
+
+  def predict(self,
+              input_data: List[str],
+              inference_parameters: Dict[str, Union[bool, str, float, int]] = {}
+             ) -> Iterable[ClassifierOutput]:
+    """ Custom prediction function for `text-classifier` model.
+
+    Args:
+      input_data (List[str]): List of text
+      inference_parameters (Dict[str, Union[bool, str, float, int]]): your inference parameters
+
+    Returns:
+      list of ClassifierOutput
+    """
+    raise NotImplementedError
+
+  @triton_wrapper.text_classifier
+  def _tritonserver_predict(self,
+                            input_data,
+                            inference_parameters: Dict[str, Union[bool, str, float, int]] = {}):
+    return self.predict(input_data, inference_parameters=inference_parameters)
+
+
+class TextEmbedder(_BaseClarifaiModel):
+  _config: ModelConfigClass = get_model_config(ModelTypes.text_embedder)
+
+  def predict(self,
+              input_data: List[str],
+              inference_parameters: Dict[str, Union[bool, str, float, int]] = {}
+             ) -> Iterable[EmbeddingOutput]:
+    """ Custom prediction function for `text-embedder` model.
+
+    Args:
+      input_data (List[str]): List of text
+      inference_parameters (Dict[str, Union[bool, str, float, int]]): your inference parameters
+
+    Returns:
+      list of EmbeddingOutput
+    """
+    raise NotImplementedError
+
+  @triton_wrapper.text_embedder
+  def _tritonserver_predict(self,
+                            input_data,
+                            inference_parameters: Dict[str, Union[bool, str, float, int]] = {}):
+    return self.predict(input_data, inference_parameters=inference_parameters)
+
+
+class TextToImage(_BaseClarifaiModel):
+  _config: ModelConfigClass = get_model_config(ModelTypes.text_to_image)
+
+  def predict(
+      self,
+      input_data: List[str],
+      inference_parameters: Dict[str, Union[bool, str, float, int]] = {}) -> Iterable[ImageOutput]:
+    """ Custom prediction function for `text-to-image` model.
+
+    Args:
+      input_data (List[str]): List of text
+      inference_parameters (Dict[str, Union[bool, str, float, int]]): your inference parameters
+
+    Returns:
+      list of ImageOutput
+    """
+    raise NotImplementedError
+
+  @triton_wrapper.text_to_image
+  def _tritonserver_predict(self,
+                            input_data,
+                            inference_parameters: Dict[str, Union[bool, str, float, int]] = {}):
+    return self.predict(input_data, inference_parameters=inference_parameters)
+
+
+class TextToText(_BaseClarifaiModel):
+  _config: ModelConfigClass = get_model_config(ModelTypes.text_to_text)
+
+  def predict(
+      self,
+      input_data: List[str],
+      inference_parameters: Dict[str, Union[bool, str, float, int]] = {}) -> Iterable[TextOutput]:
+    """ Custom prediction function for `text-to-text` (also called as `text generation`) model.
+
+    Args:
+      input_data (List[str]): List of text
+      inference_parameters (Dict[str, Union[bool, str, float, int]]): your inference parameters
+
+    Returns:
+      list of TextOutput
+    """
+    raise NotImplementedError
+
+  @triton_wrapper.text_to_text
+  def _tritonserver_predict(self,
+                            input_data,
+                            inference_parameters: Dict[str, Union[bool, str, float, int]] = {}):
+    """ This method is invoked within tritonserver, specifically in the model.py of the Python backend. Attempting to execute it outside of the triton environment will result in failure."""
+    return self.predict(input_data, inference_parameters=inference_parameters)
+
+
+class VisualClassifier(_BaseClarifaiModel):
+  _config: ModelConfigClass = get_model_config(ModelTypes.visual_classifier)
+
+  def predict(self,
+              input_data: List[np.ndarray],
+              inference_parameters: Dict[str, Union[bool, str, float, int]] = {}
+             ) -> Iterable[ClassifierOutput]:
+    """ Custom prediction function for `visual-classifier` model.
+
+    Args:
+      input_data (List[np.ndarray]): List of image
+      inference_parameters (Dict[str, Union[bool, str, float, int]]): your inference parameters
+
+    Returns:
+      list of ClassifierOutput
+    """
+    raise NotImplementedError
+
+  @triton_wrapper.visual_classifier
+  def _tritonserver_predict(self,
+                            input_data,
+                            inference_parameters: Dict[str, Union[bool, str, float, int]] = {}):
+    """ This method is invoked within tritonserver, specifically in the model.py of the Python backend. Attempting to execute it outside of the triton environment will result in failure."""
+    return self.predict(input_data, inference_parameters=inference_parameters)
+
+
+class VisualDetector(_BaseClarifaiModel):
+  _config: ModelConfigClass = get_model_config(ModelTypes.visual_detector)
+
+  def predict(self,
+              input_data: List[np.ndarray],
+              inference_parameters: Dict[str, Union[bool, str, float, int]] = {}
+             ) -> Iterable[VisualDetectorOutput]:
+    """ Custom prediction function for `visual-detector` model.
+
+    Args:
+      input_data (List[np.ndarray]): List of image
+      inference_parameters (Dict[str, Union[bool, str, float, int]]): your inference parameters
+
+    Returns:
+      list of VisualDetectorOutput
+    """
+    raise NotImplementedError
+
+  @triton_wrapper.visual_detector
+  def _tritonserver_predict(self,
+                            input_data,
+                            inference_parameters: Dict[str, Union[bool, str, float, int]] = {}):
+    """ This method is invoked within tritonserver, specifically in the model.py of the Python backend. Attempting to execute it outside of the triton environment will result in failure."""
+    return self.predict(input_data, inference_parameters=inference_parameters)
+
+  @staticmethod
+  def postprocess(width: int,
+                  height: int,
+                  labels: list,
+                  scores: list,
+                  xyxy_boxes: list,
+                  max_bbox_count: int = 500) -> VisualDetectorOutput:
+    """Convert detection output to Clarifai detector output format
+
+    Args:
+        width (int): image width
+        height (int): image height
+        labels (list): list of labels
+        scores (list): list of scores
+        xyxy_boxes (list): list of bounding boxes in x_min, y_min, x_max, y_max format
+        max_bbox_count (int, optional): Maximum detection result. Defaults to 500.
+
+    Returns:
+        VisualDetectorOutput
+    """
+    assert len(labels) == len(scores) == len(
+        xyxy_boxes
+    ), f"Length of `labels`, `scores` and `bboxes` must be equal, got {len(labels)}, {len(scores)} and {len(xyxy_boxes)} "
+    labels = [[each] for each in labels]
+    scores = [[each] for each in scores]
+    bboxes = [[x[1] / height, x[0] / width, x[3] / height, x[2] / width]
+              for x in xyxy_boxes]  # normalize the bboxes to [0,1] and [y1 x1 y2 x2]
+    bboxes = np.clip(bboxes, 0, 1.)
+    if len(bboxes) != 0:
+      bboxes = np.concatenate((bboxes, np.zeros((max_bbox_count - len(bboxes), 4))))
+      scores = np.concatenate((scores, np.zeros((max_bbox_count - len(scores), 1))))
+      labels = np.concatenate((labels, np.zeros((max_bbox_count - len(labels), 1),
+                                                dtype=np.int32)))
+    else:
+      bboxes = np.zeros((max_bbox_count, 4), dtype=np.float32)
+      scores = np.zeros((max_bbox_count, 1), dtype=np.float32)
+      labels = np.zeros((max_bbox_count, 1), dtype=np.int32)
+
+    output = VisualDetectorOutput(
+        predicted_bboxes=bboxes, predicted_labels=labels, predicted_scores=scores)
+
+    return output
+
+
+class VisualEmbedder(_BaseClarifaiModel):
+  _config: ModelConfigClass = get_model_config(ModelTypes.visual_embedder)
+
+  def predict(self,
+              input_data: List[np.ndarray],
+              inference_parameters: Dict[str, Union[bool, str, float, int]] = {}
+             ) -> Iterable[EmbeddingOutput]:
+    """ Custom prediction function for `visual-embedder` model.
+
+    Args:
+      input_data (List[np.ndarray]): List of image
+      inference_parameters (Dict[str, Union[bool, str, float, int]]): your inference parameters
+
+    Returns:
+      list of EmbeddingOutput
+    """
+    raise NotImplementedError
+
+  @triton_wrapper.visual_embedder
+  def _tritonserver_predict(self,
+                            input_data,
+                            inference_parameters: Dict[str, Union[bool, str, float, int]] = {}):
+    """ This method is invoked within tritonserver, specifically in the model.py of the Python backend. Attempting to execute it outside of the triton environment will result in failure."""
+    return self.predict(input_data, inference_parameters=inference_parameters)
+
+
+class VisualSegmenter(_BaseClarifaiModel):
+  _config: ModelConfigClass = get_model_config(ModelTypes.visual_segmenter)
+
+  def predict(
+      self,
+      input_data: List[np.ndarray],
+      inference_parameters: Dict[str, Union[bool, str, float, int]] = {}) -> Iterable[MasksOutput]:
+    """ Custom prediction function for `visual-segmenter` model.
+
+    Args:
+      input_data (List[np.ndarray]): List of image
+      inference_parameters (Dict[str, Union[bool, str, float, int]]): your inference parameters
+
+    Returns:
+      list of MasksOutput
+    """
+    raise NotImplementedError
+
+  @triton_wrapper.visual_segmenter
+  def _tritonserver_predict(self,
+                            input_data,
+                            inference_parameters: Dict[str, Union[bool, str, float, int]] = {}):
+    """ This method is invoked within tritonserver, specifically in the model.py of the Python backend. Attempting to execute it outside of the triton environment will result in failure."""
+    return self.predict(input_data, inference_parameters=inference_parameters)