nexaai 0.0.8.1__tar.gz → 0.0.8.2__tar.gz

{nexaai-0.0.8.1 → nexaai-0.0.8.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: nexaai
-Version: 0.0.8.1
+Version: 0.0.8.2
 Summary: Nexa AI SDK
 Author-Email: Nexa AI <octopus@nexa4ai.com>
 License: MIT
@@ -28,10 +28,9 @@ Requires-Dist: pillow
 Requires-Dist: prompt_toolkit
 Requires-Dist: tqdm
 Requires-Dist: tabulate
-Requires-Dist: streamlit
+Requires-Dist: streamlit>=1.37.1
 Requires-Dist: streamlit-audiorec
 Requires-Dist: python-multipart
-Requires-Dist: streamlit-audiorec
 Requires-Dist: cmake
 Requires-Dist: librosa; extra == "onnx"
 Requires-Dist: optimum[onnxruntime]; extra == "onnx"
@@ -70,7 +69,7 @@ Description-Content-Type: text/markdown
 
 </div>
 
-Nexa SDK is a comprehensive toolkit for supporting **ONNX** and **GGML** models. It supports text generation, image generation, vision-language models (VLM), and text-to-speech (TTS) capabilities. Additionally, it offers an OpenAI-compatible API server with JSON schema mode for function calling and streaming support, and a user-friendly Streamlit UI.
+Nexa SDK is a comprehensive toolkit for supporting **ONNX** and **GGML** models. It supports text generation, image generation, vision-language models (VLM), and text-to-speech (TTS) capabilities. Additionally, it offers an OpenAI-compatible API server with JSON schema mode for function calling and streaming support, and a user-friendly Streamlit UI. Users can run Nexa SDK in any device with Python environment, and GPU acceleration is supported.
 
 ## Features
 
@@ -111,6 +110,17 @@ Below is our differentiation from other similar tools:
 
 We have released pre-built wheels for various Python versions, platforms, and backends for convenient installation on our [index page](https://nexaai.github.io/nexa-sdk/whl/).
 
+How to clone this repo
+```bash
+git clone --recursive https://github.com/NexaAI/nexa-sdk
+```
+
+If you forget to use `--recursive`, you can use below command to add submodule
+```bash
+git submodule update --init --recursive
+```
+
+
 #### CPU
 
 ```bash
@@ -142,14 +152,31 @@ CMAKE_ARGS="-DGGML_METAL=ON -DSD_METAL=ON" pip install nexaai --prefer-binary --
 
 #### GPU (CUDA)
 
-For the GPU version supporting **CUDA (Linux/Windows)**:
+For **Linux**:
 
 ```bash
 CMAKE_ARGS="-DGGML_CUDA=ON -DSD_CUBLAS=ON" pip install nexaai --prefer-binary --index-url https://nexaai.github.io/nexa-sdk/whl/cu124 --extra-index-url https://pypi.org/simple --no-cache-dir
 ```
 
+For **Windows PowerShell**:
+
+```bash
+$env:CMAKE_ARGS="-DGGML_CUDA=ON -DSD_CUBLAS=ON"; pip install nexaai --prefer-binary --index-url https://nexaai.github.io/nexa-sdk/whl/cu124 --extra-index-url https://pypi.org/simple --no-cache-dir
+```
+
+For **Windows Command Prompt**:
+
+```bash
+set CMAKE_ARGS="-DGGML_CUDA=ON -DSD_CUBLAS=ON" & pip install nexaai --prefer-binary --index-url https://nexaai.github.io/nexa-sdk/whl/cu124 --extra-index-url https://pypi.org/simple --no-cache-dir
+```
+
+For **Windows Git Bash**:
+
+```bash
+CMAKE_ARGS="-DGGML_CUDA=ON -DSD_CUBLAS=ON" pip install nexaai --prefer-binary --index-url https://nexaai.github.io/nexa-sdk/whl/cu124 --extra-index-url https://pypi.org/simple --no-cache-dir
+```
 > [!NOTE]
-> The CUDA wheels are built with CUDA 12.4, but should be compatible with all CUDA 12.X
+> If you want to use ONNX model, just replace `pip install nexaai` with `pip install nexaai[onnx]` in above commands
 
 
 <details>
@@ -259,4 +286,4 @@ For detailed information on server setup, API endpoints, and usage examples, ple
 We would like to thank the following projects:
 - [llama.cpp](https://github.com/ggerganov/llama.cpp)
 - [stable-diffusion.cpp](https://github.com/leejet/stable-diffusion.cpp)
-- [optimum](https://github.com/huggingface/optimum)
+- [optimum](https://github.com/huggingface/optimum)

{nexaai-0.0.8.1 → nexaai-0.0.8.2}/README.md

@@ -21,7 +21,7 @@
 
 </div>
 
-Nexa SDK is a comprehensive toolkit for supporting **ONNX** and **GGML** models. It supports text generation, image generation, vision-language models (VLM), and text-to-speech (TTS) capabilities. Additionally, it offers an OpenAI-compatible API server with JSON schema mode for function calling and streaming support, and a user-friendly Streamlit UI.
+Nexa SDK is a comprehensive toolkit for supporting **ONNX** and **GGML** models. It supports text generation, image generation, vision-language models (VLM), and text-to-speech (TTS) capabilities. Additionally, it offers an OpenAI-compatible API server with JSON schema mode for function calling and streaming support, and a user-friendly Streamlit UI. Users can run Nexa SDK in any device with Python environment, and GPU acceleration is supported.
 
 ## Features
 
@@ -62,6 +62,17 @@ Below is our differentiation from other similar tools:
 
 We have released pre-built wheels for various Python versions, platforms, and backends for convenient installation on our [index page](https://nexaai.github.io/nexa-sdk/whl/).
 
+How to clone this repo
+```bash
+git clone --recursive https://github.com/NexaAI/nexa-sdk
+```
+
+If you forget to use `--recursive`, you can use below command to add submodule
+```bash
+git submodule update --init --recursive
+```
+
+
 #### CPU
 
 ```bash
@@ -93,14 +104,31 @@ CMAKE_ARGS="-DGGML_METAL=ON -DSD_METAL=ON" pip install nexaai --prefer-binary --
 
 #### GPU (CUDA)
 
-For the GPU version supporting **CUDA (Linux/Windows)**:
+For **Linux**:
 
 ```bash
 CMAKE_ARGS="-DGGML_CUDA=ON -DSD_CUBLAS=ON" pip install nexaai --prefer-binary --index-url https://nexaai.github.io/nexa-sdk/whl/cu124 --extra-index-url https://pypi.org/simple --no-cache-dir
 ```
 
+For **Windows PowerShell**:
+
+```bash
+$env:CMAKE_ARGS="-DGGML_CUDA=ON -DSD_CUBLAS=ON"; pip install nexaai --prefer-binary --index-url https://nexaai.github.io/nexa-sdk/whl/cu124 --extra-index-url https://pypi.org/simple --no-cache-dir
+```
+
+For **Windows Command Prompt**:
+
+```bash
+set CMAKE_ARGS="-DGGML_CUDA=ON -DSD_CUBLAS=ON" & pip install nexaai --prefer-binary --index-url https://nexaai.github.io/nexa-sdk/whl/cu124 --extra-index-url https://pypi.org/simple --no-cache-dir
+```
+
+For **Windows Git Bash**:
+
+```bash
+CMAKE_ARGS="-DGGML_CUDA=ON -DSD_CUBLAS=ON" pip install nexaai --prefer-binary --index-url https://nexaai.github.io/nexa-sdk/whl/cu124 --extra-index-url https://pypi.org/simple --no-cache-dir
+```
 > [!NOTE]
-> The CUDA wheels are built with CUDA 12.4, but should be compatible with all CUDA 12.X
+> If you want to use ONNX model, just replace `pip install nexaai` with `pip install nexaai[onnx]` in above commands
 
 
 <details>
@@ -210,4 +238,4 @@ For detailed information on server setup, API endpoints, and usage examples, ple
 We would like to thank the following projects:
 - [llama.cpp](https://github.com/ggerganov/llama.cpp)
 - [stable-diffusion.cpp](https://github.com/leejet/stable-diffusion.cpp)
-- [optimum](https://github.com/huggingface/optimum)
+- [optimum](https://github.com/huggingface/optimum)

nexaai-0.0.8.2/nexa/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.0.8.2"

{nexaai-0.0.8.1 → nexaai-0.0.8.2}/nexa/cli/entry.py

@@ -15,27 +15,32 @@ def run_ggml_inference(args):
     
     stop_words = kwargs.pop("stop_words", [])
 
-    if run_type == "NLP":
-        from nexa.gguf.nexa_inference_text import NexaTextInference
-        inference = NexaTextInference(model_path=model_path, local_path=local_path, stop_words=stop_words, **kwargs)
-    elif run_type == "Computer Vision":
-        from nexa.gguf.nexa_inference_image import NexaImageInference
-        inference = NexaImageInference(model_path=model_path, local_path=local_path, **kwargs)
-        if hasattr(args, 'streamlit') and args.streamlit:
-            inference.run_streamlit(model_path)
-        elif args.img2img:
-            inference.run_img2img()
+    try:
+        if run_type == "NLP":
+            from nexa.gguf.nexa_inference_text import NexaTextInference
+            inference = NexaTextInference(model_path=model_path, local_path=local_path, stop_words=stop_words, **kwargs)
+        elif run_type == "Computer Vision":
+            from nexa.gguf.nexa_inference_image import NexaImageInference
+            inference = NexaImageInference(model_path=model_path, local_path=local_path, **kwargs)
+            if hasattr(args, 'streamlit') and args.streamlit:
+                inference.run_streamlit(model_path)
+            elif args.img2img:
+                inference.run_img2img()
+            else:
+                inference.run_txt2img()
+            return
+        elif run_type == "Multimodal":
+            from nexa.gguf.nexa_inference_vlm import NexaVLMInference
+            inference = NexaVLMInference(model_path=model_path, local_path=local_path, stop_words=stop_words, **kwargs)
+        elif run_type == "Audio":
+            from nexa.gguf.nexa_inference_voice import NexaVoiceInference
+            inference = NexaVoiceInference(model_path=model_path, local_path=local_path, **kwargs)
         else:
-            inference.run_txt2img()
+            print(f"Unknown task: {run_type}. Skipping inference.")
+            return
+    except Exception as e:
+        print(f"Error loading GGUF models, please refer to our docs to install nexaai package: https://docs.nexaai.com/getting-started/installation ")
         return
-    elif run_type == "Multimodal":
-        from nexa.gguf.nexa_inference_vlm import NexaVLMInference
-        inference = NexaVLMInference(model_path=model_path, local_path=local_path, stop_words=stop_words, **kwargs)
-    elif run_type == "Audio":
-        from nexa.gguf.nexa_inference_voice import NexaVoiceInference
-        inference = NexaVoiceInference(model_path=model_path, local_path=local_path, **kwargs)
-    else:
-        raise ValueError(f"Unknown task: {run_type}")
 
     if hasattr(args, 'streamlit') and args.streamlit:
         inference.run_streamlit(model_path)
@@ -49,20 +54,25 @@ def run_onnx_inference(args):
     from nexa.general import pull_model
     local_path, run_type = pull_model(model_path)
 
-    if run_type == "NLP":
-        from nexa.onnx.nexa_inference_text import NexaTextInference as NexaTextOnnxInference
-        inference = NexaTextOnnxInference(model_path=model_path, local_path=local_path, **kwargs)
-    elif run_type == "Computer Vision":
-        from nexa.onnx.nexa_inference_image import NexaImageInference as NexaImageOnnxInference
-        inference = NexaImageOnnxInference(model_path=model_path, local_path=local_path, **kwargs)
-    elif run_type == "Audio":
-        from nexa.onnx.nexa_inference_voice import NexaVoiceInference as NexaVoiceOnnxInference
-        inference = NexaVoiceOnnxInference(model_path=model_path, local_path=local_path, **kwargs)
-    elif run_type == "TTS":
-        from nexa.onnx.nexa_inference_tts import NexaTTSInference as NexaTTSOnnxInference
-        inference = NexaTTSOnnxInference(model_path=model_path, local_path=local_path, **kwargs)
-    else:
-        raise ValueError(f"Unknown task: {run_type}")
+    try:
+        if run_type == "NLP":
+            from nexa.onnx.nexa_inference_text import NexaTextInference as NexaTextOnnxInference
+            inference = NexaTextOnnxInference(model_path=model_path, local_path=local_path, **kwargs)
+        elif run_type == "Computer Vision":
+            from nexa.onnx.nexa_inference_image import NexaImageInference as NexaImageOnnxInference
+            inference = NexaImageOnnxInference(model_path=model_path, local_path=local_path, **kwargs)
+        elif run_type == "Audio":
+            from nexa.onnx.nexa_inference_voice import NexaVoiceInference as NexaVoiceOnnxInference
+            inference = NexaVoiceOnnxInference(model_path=model_path, local_path=local_path, **kwargs)
+        elif run_type == "TTS":
+            from nexa.onnx.nexa_inference_tts import NexaTTSInference as NexaTTSOnnxInference
+            inference = NexaTTSOnnxInference(model_path=model_path, local_path=local_path, **kwargs)
+        else:
+            print(f"Unknown task: {run_type}. Skipping inference.")
+            return
+    except Exception as e:
+        print(f"Error loading ONNX models, please refer to our docs to install nexaai[onnx] package: https://docs.nexaai.com/getting-started/installation ")
+        return
 
     if hasattr(args, 'streamlit') and args.streamlit:
         inference.run_streamlit(model_path)

{nexaai-0.0.8.1 → nexaai-0.0.8.2}/nexa/constants.py

@@ -120,7 +120,7 @@ NEXA_RUN_MODEL_MAP_FUNCTION_CALLING = {
 
 NEXA_RUN_PROJECTOR_MAP = {
     "nanollava": "nanoLLaVA:projector-fp16",
-    "nanoLLaVA:fp16": "nanoLLaVA:project-fp16",
+    "nanoLLaVA:fp16": "nanoLLaVA:projector-fp16",
     "llava-phi3": "llava-phi-3-mini:projector-q4_0",
     "llava-phi-3-mini:q4_0": "llava-phi-3-mini:projector-q4_0",
     "llava-phi-3-mini:fp16": "llava-phi-3-mini:projector-fp16",

{nexaai-0.0.8.1 → nexaai-0.0.8.2}/nexa/gguf/llama/llama.py

@@ -1337,22 +1337,25 @@ class Llama:
                         returned_tokens += 1
                         yield {
                             "id": completion_id,
-                            "object": "text_completion",
-                            "created": created,
                             "model": model_name,
+                            "object": "text_completion.chunk",
+                            "created": created,
                             "choices": [
                                 {
-                                    "text": self.detokenize(
-                                        [token],
-                                        prev_tokens=prompt_tokens
-                                        + completion_tokens[:returned_tokens],
-                                    ).decode("utf-8", errors="ignore"),
                                     "index": 0,
-                                    "logprobs": logprobs_or_none,
+                                    "delta": {
+                                        "content": self.detokenize(
+                                            [token],
+                                            prev_tokens=prompt_tokens
+                                            + completion_tokens[:returned_tokens],
+                                        ).decode("utf-8", errors="ignore")
+                                    },
                                     "finish_reason": None,
+                                    "logprobs": logprobs_or_none,
                                 }
                             ],
                         }
+
                 else:
                     while len(remaining_tokens) > 0:
                         decode_success = False
@@ -1383,19 +1386,22 @@ class Llama:
 
                         yield {
                             "id": completion_id,
-                            "object": "text_completion",
-                            "created": created,
                             "model": model_name,
+                            "object": "text_completion.chunk",
+                            "created": created,
                             "choices": [
                                 {
-                                    "text": ts,
                                     "index": 0,
                                     "logprobs": None,
+                                    "delta": {
+                                        "content": ts,
+                                    },
                                     "finish_reason": None,
                                 }
                             ],
                         }
 
+
             if len(completion_tokens) >= max_tokens:
                 text = self.detokenize(completion_tokens, prev_tokens=prompt_tokens)
                 finish_reason = "length"
@@ -1475,52 +1481,59 @@ class Llama:
                     returned_tokens += 1
                     yield {
                         "id": completion_id,
-                        "object": "text_completion",
-                        "created": created,
                         "model": model_name,
+                        "object": "text_completion.chunk",
+                        "created": created,
                         "choices": [
                             {
-                                "text": last_text[
-                                    : len(last_text) - (token_end_position - end)
-                                ].decode("utf-8", errors="ignore"),
                                 "index": 0,
+                                "delta": {
+                                    "content": last_text[: len(last_text) - (token_end_position - end)].decode(
+                                        "utf-8", errors="ignore"
+                                    )
+                                },
                                 "logprobs": logprobs_or_none,
                                 "finish_reason": None,
                             }
                         ],
                     }
+
                     break
                 returned_tokens += 1
                 yield {
                     "id": completion_id,
-                    "object": "text_completion",
-                    "created": created,
                     "model": model_name,
+                    "object": "text_completion.chunk",
+                    "created": created,
                     "choices": [
                         {
-                            "text": self.detokenize([token]).decode(
-                                "utf-8", errors="ignore"
-                            ),
                             "index": 0,
+                            "delta": {
+                                "content": self.detokenize([token]).decode("utf-8", errors="ignore")
+                            },
                             "logprobs": logprobs_or_none,
                             "finish_reason": None,
                         }
                     ],
                 }
+
             yield {
                 "id": completion_id,
-                "object": "text_completion",
-                "created": created,
                 "model": model_name,
+                "object": "text_completion.chunk",
+                "created": created,
                 "choices": [
                     {
-                        "text": "",
                         "index": 0,
+                        "delta": {
+                            "content": "",
+                        },
                         "logprobs": None,
                         "finish_reason": finish_reason,
                     }
                 ],
             }
+
             if self.cache:
                 if self.verbose:
                     print("Llama._create_completion: cache save", file=sys.stderr)

{nexaai-0.0.8.1 → nexaai-0.0.8.2}/nexa/gguf/llama/llama_chat_format.py

@@ -304,6 +304,7 @@ def _convert_text_completion_chunks_to_chat(
                     }
                 ],
             }
+
         yield {
             "id": "chat" + chunk["id"],
             "model": chunk["model"],
@@ -314,7 +315,7 @@ def _convert_text_completion_chunks_to_chat(
                     "index": 0,
                     "delta": (
                         {
-                            "content": chunk["choices"][0]["text"],
+                            "content": chunk["choices"][0]["delta"]["content"],
                         }
                         if chunk["choices"][0]["finish_reason"] is None
                         else {}

{nexaai-0.0.8.1 → nexaai-0.0.8.2}/nexa/gguf/llama/llama_types.py

@@ -278,6 +278,24 @@ ChatCompletionToolChoiceOption = Union[
     Literal["none", "auto", "required"], ChatCompletionNamedToolChoice
 ]
 
+class CompletionStreamResponseDelta(TypedDict):
+    content: Optional[str]
+
+
+class CompletionStreamResponseChoice(TypedDict):
+    index: int
+    delta: CompletionStreamResponseDelta
+    finish_reason: Optional[Literal["stop", "length"]]
+    logprobs: NotRequired[Optional[CompletionLogprobs]]
+
+
+class CreateCompletionStreamResponse(TypedDict):
+    id: str
+    model: str
+    object: Literal["text_completion.chunk"]
+    created: int
+    choices: List[CompletionStreamResponseChoice]
+
 
 # NOTE: The following type names are not part of the OpenAI OpenAPI specification
 # and will be removed in a future major release.
@@ -285,7 +303,6 @@ ChatCompletionToolChoiceOption = Union[
 EmbeddingData = Embedding
 CompletionChunk = CreateCompletionResponse
 Completion = CreateCompletionResponse
-CreateCompletionStreamResponse = CreateCompletionResponse
 ChatCompletionMessage = ChatCompletionResponseMessage
 ChatCompletionChoice = ChatCompletionResponseChoice
 ChatCompletion = CreateChatCompletionResponse

{nexaai-0.0.8.1 → nexaai-0.0.8.2}/nexa/gguf/nexa_inference_image.py

@@ -115,7 +115,7 @@ class NexaImageInference:
             except Exception as e:
                 logging.error(f"Attempt {attempt + 1} failed with error: {e}")
                 time.sleep(1)
-        logging.error("All retry attempts failed.")
+        print("All retry attempts failed becase of Out of Memory error, Try to use smaller models...")
         return None
 
     def txt2img(

{nexaai-0.0.8.1 → nexaai-0.0.8.2}/nexa/gguf/streamlit/streamlit_image_chat.py

@@ -4,6 +4,7 @@ from PIL import Image
 from nexa.general import pull_model
 import streamlit as st
 from nexa.gguf.nexa_inference_image import NexaImageInference
+import io
 
 default_model = sys.argv[1]
 
@@ -106,5 +107,16 @@ if st.button("Generate Image"):
                 st.session_state.nexa_model, prompt, negative_prompt
             )
             st.success("Images generated successfully!")
-            for image in images:
-                st.image(image, caption="Generated Image", use_column_width=True)
+            for i, image in enumerate(images):
+                st.image(image, caption=f"Generated Image", use_column_width=True)
+                
+                img_byte_arr = io.BytesIO()
+                image.save(img_byte_arr, format='PNG')
+                img_byte_arr = img_byte_arr.getvalue()
+
+                st.download_button(
+                    label=f"Download Image",
+                    data=img_byte_arr,
+                    file_name=f"generated_image.png",
+                    mime="image/png"
+                )

{nexaai-0.0.8.1 → nexaai-0.0.8.2}/nexa/onnx/streamlit/streamlit_image_chat.py

@@ -3,7 +3,7 @@ import sys
 
 import numpy as np
 import streamlit as st
-
+from optimum.onnxruntime import ORTLatentConsistencyModelPipeline
 from nexa.general import pull_model
 from nexa.onnx.nexa_inference_image import NexaImageInference
 
@@ -15,11 +15,11 @@ def load_model(model_path):
     local_path, run_type = pull_model(model_path)
     nexa_model = NexaImageInference(model_path=model_path, local_path=local_path)
 
-    if nexa_model.downloaded_onnx_folder is None:
+    if nexa_model.download_onnx_folder is None:
         st.error("Failed to download the model. Please check the model path.")
         return None
 
-    nexa_model._load_model(nexa_model.downloaded_onnx_folder)
+    nexa_model._load_model(nexa_model.download_onnx_folder)
     return nexa_model
 
 
@@ -30,17 +30,21 @@ def generate_images(nexa_model: NexaImageInference, prompt, negative_prompt):
 
     generator = np.random.RandomState(nexa_model.params["random_seed"])
 
-    images = nexa_model.pipeline(
-        prompt=prompt,
-        negative_prompt=negative_prompt if negative_prompt else None,
-        num_inference_steps=nexa_model.params["num_inference_steps"],
-        num_images_per_prompt=nexa_model.params["num_images_per_prompt"],
-        height=nexa_model.params["height"],
-        width=nexa_model.params["width"],
-        generator=generator,
-        guidance_scale=nexa_model.params["guidance_scale"],
-    ).images
+    is_lcm_pipeline = isinstance(nexa_model.pipeline, ORTLatentConsistencyModelPipeline)
+
+    pipeline_kwargs = {
+        "prompt": prompt,
+        "num_inference_steps": nexa_model.params["num_inference_steps"],
+        "num_images_per_prompt": nexa_model.params["num_images_per_prompt"],
+        "height": nexa_model.params["height"],
+        "width": nexa_model.params["width"],
+        "generator": generator,
+        "guidance_scale": nexa_model.params["guidance_scale"],
+    }
+    if not is_lcm_pipeline and negative_prompt:
+        pipeline_kwargs["negative_prompt"] = negative_prompt
 
+    images = nexa_model.pipeline(**pipeline_kwargs).images
     return images
 
 

{nexaai-0.0.8.1 → nexaai-0.0.8.2}/pyproject.toml

@@ -24,10 +24,9 @@ dependencies = [
     "prompt_toolkit",
     "tqdm",                     # Shared dependencies
     "tabulate",
-    "streamlit",
+    "streamlit>=1.37.1",
     "streamlit-audiorec",
     "python-multipart",
-    "streamlit-audiorec",
     "cmake",
 ]
 classifiers = [