nexaai 0.0.8.7__tar.gz → 0.0.8.8__tar.gz

{nexaai-0.0.8.7 → nexaai-0.0.8.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: nexaai
-Version: 0.0.8.7
+Version: 0.0.8.8
 Summary: Nexa AI SDK
 Author-Email: Nexa AI <octopus@nexa4ai.com>
 License: MIT
@@ -53,6 +53,13 @@ Requires-Dist: pytablewriter; extra == "eval"
 Requires-Dist: sacrebleu; extra == "eval"
 Requires-Dist: langdetect; extra == "eval"
 Requires-Dist: immutabledict; extra == "eval"
+Requires-Dist: hydra-core; extra == "eval"
+Requires-Dist: psutil; extra == "eval"
+Requires-Dist: typing-extensions; extra == "eval"
+Requires-Dist: flatten_dict; extra == "eval"
+Requires-Dist: colorlog; extra == "eval"
+Requires-Dist: pandas; extra == "eval"
+Requires-Dist: rich; extra == "eval"
 Provides-Extra: onnx
 Provides-Extra: eval
 Description-Content-Type: text/markdown
@@ -83,10 +90,18 @@ Description-Content-Type: text/markdown
 Nexa SDK is a comprehensive toolkit for supporting **ONNX** and **GGML** models. It supports text generation, image generation, vision-language models (VLM), and speech-to-text (ASR), and text-to-speech (TTS) capabilities. Additionally, it offers an OpenAI-compatible API server with JSON schema mode for function calling and streaming support, and a user-friendly Streamlit UI. Users can run Nexa SDK in any device with Python environment, and GPU acceleration is supported, including CUDA, Metal, and ROCm. An executable version is also available.
 
 ## Latest News 🔥
-* [2024/09] Nexa now has executables for easy installation: [Install Nexa SDK](https://nexaai.com/download-sdk)
+* [2024/10] Support embedding model: `nexa embed <model_path> <prompt>`
+* [2024/10] Support pull and run supported Computer Vision models in GGUF format from HuggingFace: `nexa run -hf <model_id> -mt COMPUTER_VISION`
+* [2024/10] Support VLM in local server.
+* [2024/10] Added option to customize maximum context window for NLP and VLM models.
+* [2024/10] Support running model from user's local path
+* [2024/10] Added LoRA support for NLP models. 
+* [2024/10] Added support for whisper-large-v3-turbo: `nexa run faster-whisper-large-turbo`
+* [2024/10] Added support for AMD-Llama-135m: `nexa run AMD-Llama-135m:fp16`
+* [2024/09] Nexa now has executables for easy installation: [Install Nexa SDK](https://nexaai.com/download-sdk) ✨
 * [2024/09] Added support for Llama 3.2 models: `nexa run llama3.2`
 * [2024/09] Added support for Qwen2.5, Qwen2.5-coder and Qwen2.5-Math models: `nexa run qwen2.5`
-* [2024/09] Now supporting pulling and running GGUF models from Hugging Face: `nexa run -hf <hf model id>`
+* [2024/09] Support pull and run NLP models in GGUF format from HuggingFace: `nexa run -hf <model_id> -mt NLP`
 * [2024/09] Added support for ROCm
 * [2024/09] Added support for Phi-3.5 models: `nexa run phi3.5`
 * [2024/09] Added support for OpenELM models: `nexa run openelm`
@@ -252,60 +267,66 @@ Below is our differentiation from other similar tools:
 ## Supported Models & Model Hub
 Our on-device model hub offers all types of quantized models (text, image, audio, multimodal) with filters for RAM, file size, Tasks, etc. to help you easily explore models with UI. Explore on-device models at [On-device Model Hub](https://model-hub.nexa4ai.com/)
 
-Some supported models:
-| Model                                                                                                   | Type            | Format    | Command                            |
-| ------------------------------------------------------------------------------------------------------- | --------------- | --------- | ---------------------------------- |
-| [octopus-v2](https://www.nexaai.com/NexaAI/Octopus-v2/gguf-q4_0/readme)                                 | NLP             | GGUF      | `nexa run octopus-v2`              |
-| [octopus-v4](https://www.nexaai.com/NexaAI/Octopus-v4/gguf-q4_0/readme)                                 | NLP             | GGUF      | `nexa run octopus-v4`              |
-| [gpt2](https://nexaai.com/openai/gpt2/gguf-q4_0/readme)                                                 | NLP             | GGUF      | `nexa run gpt2`                    |
-| [tinyllama](https://www.nexaai.com/TinyLlama/TinyLlama-1.1B-Chat-v1.0/gguf-fp16/readme)                 | NLP             | GGUF      | `nexa run tinyllama`               |
-| [llama2](https://www.nexaai.com/meta/Llama2-7b-chat/gguf-q4_0/readme)                                   | NLP             | GGUF/ONNX | `nexa run llama2`                  |
-| [llama2-uncensored](https://www.nexaai.com/georgesung/Llama2-7b-chat-uncensored/gguf-q4_0/readme)       | NLP             | GGUF      | `nexa run llama2-uncensored`       |
-| [llama2-function-calling](https://www.nexaai.com/Trelis/Llama2-7b-function-calling/gguf-q4_K_M/readme)  | NLP             | GGUF      | `nexa run llama2-function-calling` |
-| [llama3](https://www.nexaai.com/meta/Llama3-8B-Instruct/gguf-q4_0/readme)                               | NLP             | GGUF/ONNX | `nexa run llama3`                  |
-| [llama3.1](https://www.nexaai.com/meta/Llama3.1-8B-Instruct/gguf-q4_0/readme)                           | NLP             | GGUF/ONNX | `nexa run llama3.1`                |
-| [llama3.2](https://nexaai.com/meta/Llama3.2-3B-Instruct/gguf-q4_0/readme)                               | NLP             | GGUF      | `nexa run llama3.2`                |
-| [llama3-uncensored](https://www.nexaai.com/Orenguteng/Llama3-8B-Lexi-Uncensored/gguf-q4_K_M/readme)     | NLP             | GGUF      | `nexa run llama3-uncensored`       |
-| [gemma](https://www.nexaai.com/google/gemma-1.1-2b-instruct/gguf-q4_0/readme)                           | NLP             | GGUF/ONNX | `nexa run gemma`                   |
-| [gemma2](https://www.nexaai.com/google/gemma-2-2b-instruct/gguf-q4_0/readme)                            | NLP             | GGUF      | `nexa run gemma2`                  |
-| [qwen1.5](https://www.nexaai.com/Qwen/Qwen1.5-7B-Instruct/gguf-q4_0/readme)                             | NLP             | GGUF      | `nexa run qwen1.5`                 |
-| [qwen2](https://www.nexaai.com/Qwen/Qwen2-1.5B-Instruct/gguf-q4_0/readme)                               | NLP             | GGUF/ONNX | `nexa run qwen2`                   |
-| [qwen2.5](https://www.nexaai.com/Qwen/Qwen2.5-1.5B-Instruct/gguf-q4_0/readme)                           | NLP             | GGUF      | `nexa run qwen2.5`                 |
-| [mathqwen](https://nexaai.com/Qwen/Qwen2.5-Math-1.5B-Instruct/gguf-q4_0/readme)                         | NLP             | GGUF      | `nexa run mathqwen`                |
-| [codeqwen](https://www.nexaai.com/Qwen/CodeQwen1.5-7B-Instruct/gguf-q4_0/readme)                        | NLP             | GGUF      | `nexa run codeqwen`                |
-| [mistral](https://www.nexaai.com/mistralai/Mistral-7B-Instruct-v0.3/gguf-q4_0/readme)                   | NLP             | GGUF/ONNX | `nexa run mistral`                 |
-| [dolphin-mistral](https://www.nexaai.com/CognitiveComputations/dolphin-2.8-mistral-7b/gguf-q4_0/readme) | NLP             | GGUF      | `nexa run dolphin-mistral`         |
-| [codegemma](https://www.nexaai.com/google/codegemma-2b/gguf-q4_0/readme)                                | NLP             | GGUF      | `nexa run codegemma`               |
-| [codellama](https://www.nexaai.com/meta/CodeLlama-7b-Instruct/gguf-q2_K/readme)                         | NLP             | GGUF      | `nexa run codellama`               |
-| [deepseek-coder](https://www.nexaai.com/DeepSeek/deepseek-coder-1.3b-instruct/gguf-q4_0/readme)         | NLP             | GGUF      | `nexa run deepseek-coder`          |
-| [phi2](https://www.nexaai.com/microsoft/Phi-2/gguf-q4_0/readme)                                         | NLP             | GGUF      | `nexa run phi2`                    |
-| [phi3](https://www.nexaai.com/microsoft/Phi-3-mini-128k-instruct/gguf-q4_0/readme)                      | NLP             | GGUF/ONNX | `nexa run phi3`                    |
-| [phi3.5](https://nexaai.com/microsoft/Phi-3.5-mini-instruct/gguf-q4_0/readme)                           | NLP             | GGUF      | `nexa run phi3.5`                  |
-| [openelm](https://nexaai.com/apple/OpenELM-3B/gguf-q4_K_M/readme)                                       | NLP             | GGUF      | `nexa run openelm`                 |
-| [nanollava](https://www.nexaai.com/qnguyen3/nanoLLaVA/gguf-fp16/readme)                                 | Multimodal      | GGUF      | `nexa run nanollava`               |
-| [llava-phi3](https://www.nexaai.com/xtuner/llava-phi-3-mini/gguf-q4_0/readme)                           | Multimodal      | GGUF      | `nexa run llava-phi3`              |
-| [llava-llama3](https://www.nexaai.com/xtuner/llava-llama-3-8b-v1.1/gguf-q4_0/readme)                    | Multimodal      | GGUF      | `nexa run llava-llama3`            |
-| [llava1.6-mistral](https://www.nexaai.com/liuhaotian/llava-v1.6-mistral-7b/gguf-q4_0/readme)            | Multimodal      | GGUF      | `nexa run llava1.6-mistral`        |
-| [llava1.6-vicuna](https://www.nexaai.com/liuhaotian/llava-v1.6-vicuna-7b/gguf-q4_0/readme)              | Multimodal      | GGUF      | `nexa run llava1.6-vicuna`         |
-| [stable-diffusion-v1-4](https://www.nexaai.com/runwayml/stable-diffusion-v1-4/gguf-q4_0/readme)         | Computer Vision | GGUF      | `nexa run sd1-4`                   |
-| [stable-diffusion-v1-5](https://www.nexaai.com/runwayml/stable-diffusion-v1-5/gguf-q4_0/readme)         | Computer Vision | GGUF/ONNX | `nexa run sd1-5`                   |
-| [stable-diffusion-v2-1](https://nexaai.com/StabilityAI/stable-diffusion-v2-1/gguf-q4_0/readme)          | Computer Vision | GGUF      | `nexa run sd2-1`                   |
-| [stable-diffusion-3-medium](https://nexaai.com/StabilityAI/stable-diffusion-3-medium/gguf-q4_0/readme)  | Computer Vision | GGUF      | `nexa run sd3`                     |
-| [FLUX.1-schnell](https://nexaai.com/BlackForestLabs/FLUX.1-schnell/gguf-q4_0/readme)                    | Computer Vision | GGUF      | `nexa run flux`                    |
-| [lcm-dreamshaper](https://www.nexaai.com/SimianLuo/lcm-dreamshaper-v7/gguf-fp16/readme)                 | Computer Vision | GGUF/ONNX | `nexa run lcm-dreamshaper`         |
-| [hassaku-lcm](https://nexaai.com/stablediffusionapi/hassaku-hentai-model-v13-LCM/gguf-fp16/readme)      | Computer Vision | GGUF      | `nexa run hassaku-lcm`             |
-| [anything-lcm](https://www.nexaai.com/Linaqruf/anything-v30-LCM/gguf-fp16/readme)                       | Computer Vision | GGUF      | `nexa run anything-lcm`            |
-| [faster-whisper-tiny](https://www.nexaai.com/Systran/faster-whisper-tiny/bin-cpu-fp16/readme)           | Audio           | BIN       | `nexa run faster-whisper-tiny`     |
-| [faster-whisper-small](https://www.nexaai.com/Systran/faster-whisper-small/bin-cpu-fp16/readme)         | Audio           | BIN       | `nexa run faster-whisper-small`    |
-| [faster-whisper-medium](https://www.nexaai.com/Systran/faster-whisper-medium/bin-cpu-fp16/readme)       | Audio           | BIN       | `nexa run faster-whisper-medium`   |
-| [faster-whisper-base](https://www.nexaai.com/Systran/faster-whisper-base/bin-cpu-fp16/readme)           | Audio           | BIN       | `nexa run faster-whisper-base`     |
-| [faster-whisper-large](https://www.nexaai.com/Systran/faster-whisper-large-v3/bin-cpu-fp16/readme)      | Audio           | BIN       | `nexa run faster-whisper-large`    |
-| [whisper-tiny.en](https://nexaai.com/openai/whisper-tiny.en/onnx-cpu-fp32/readme)                       | Audio           | ONNX      | `nexa run whisper-tiny.en`         |
-| [whisper-tiny](https://nexaai.com/openai/whisper-tiny/onnx-cpu-fp32/readme)                             | Audio           | ONNX      | `nexa run whisper-tiny`            |
-| [whisper-small.en](https://nexaai.com/openai/whisper-small.en/onnx-cpu-fp32/readme)                     | Audio           | ONNX      | `nexa run whisper-small.en`        |
-| [whisper-small](https://nexaai.com/openai/whisper-small/onnx-cpu-fp32/readme)                           | Audio           | ONNX      | `nexa run whisper-small`           |
-| [whisper-base.en](https://nexaai.com/openai/whisper-base.en/onnx-cpu-fp32/readme)                       | Audio           | ONNX      | `nexa run whisper-base.en`         |
-| [whisper-base](https://nexaai.com/openai/whisper-base/onnx-cpu-fp32/readme)                             | Audio           | ONNX      | `nexa run whisper-base`            |
+Supported models (full list at [Model Hub](https://nexa.ai/models)):
+| Model                                                                                                   | Type            | Format    | Command                                |
+| ------------------------------------------------------------------------------------------------------- | --------------- | --------- | -------------------------------------- |
+| [octopus-v2](https://www.nexaai.com/NexaAI/Octopus-v2/gguf-q4_0/readme)                                 | NLP             | GGUF      | `nexa run octopus-v2`                  |
+| [octopus-v4](https://www.nexaai.com/NexaAI/Octopus-v4/gguf-q4_0/readme)                                 | NLP             | GGUF      | `nexa run octopus-v4`                  |
+| [gpt2](https://nexaai.com/openai/gpt2/gguf-q4_0/readme)                                                 | NLP             | GGUF      | `nexa run gpt2`                        |
+| [tinyllama](https://www.nexaai.com/TinyLlama/TinyLlama-1.1B-Chat-v1.0/gguf-fp16/readme)                 | NLP             | GGUF      | `nexa run tinyllama`                   |
+| [llama2](https://www.nexaai.com/meta/Llama2-7b-chat/gguf-q4_0/readme)                                   | NLP             | GGUF/ONNX | `nexa run llama2`                      |
+| [llama2-uncensored](https://www.nexaai.com/georgesung/Llama2-7b-chat-uncensored/gguf-q4_0/readme)       | NLP             | GGUF      | `nexa run llama2-uncensored`           |
+| [llama2-function-calling](https://www.nexaai.com/Trelis/Llama2-7b-function-calling/gguf-q4_K_M/readme)  | NLP             | GGUF      | `nexa run llama2-function-calling`     |
+| [llama3](https://www.nexaai.com/meta/Llama3-8B-Instruct/gguf-q4_0/readme)                               | NLP             | GGUF/ONNX | `nexa run llama3`                      |
+| [llama3.1](https://www.nexaai.com/meta/Llama3.1-8B-Instruct/gguf-q4_0/readme)                           | NLP             | GGUF/ONNX | `nexa run llama3.1`                    |
+| [llama3.2](https://nexaai.com/meta/Llama3.2-3B-Instruct/gguf-q4_0/readme)                               | NLP             | GGUF      | `nexa run llama3.2`                    |
+| [llama3-uncensored](https://www.nexaai.com/Orenguteng/Llama3-8B-Lexi-Uncensored/gguf-q4_K_M/readme)     | NLP             | GGUF      | `nexa run llama3-uncensored`           |
+| [gemma](https://www.nexaai.com/google/gemma-1.1-2b-instruct/gguf-q4_0/readme)                           | NLP             | GGUF/ONNX | `nexa run gemma`                       |
+| [gemma2](https://www.nexaai.com/google/gemma-2-2b-instruct/gguf-q4_0/readme)                            | NLP             | GGUF      | `nexa run gemma2`                      |
+| [qwen1.5](https://www.nexaai.com/Qwen/Qwen1.5-7B-Instruct/gguf-q4_0/readme)                             | NLP             | GGUF      | `nexa run qwen1.5`                     |
+| [qwen2](https://www.nexaai.com/Qwen/Qwen2-1.5B-Instruct/gguf-q4_0/readme)                               | NLP             | GGUF/ONNX | `nexa run qwen2`                       |
+| [qwen2.5](https://www.nexaai.com/Qwen/Qwen2.5-1.5B-Instruct/gguf-q4_0/readme)                           | NLP             | GGUF      | `nexa run qwen2.5`                     |
+| [mathqwen](https://nexaai.com/Qwen/Qwen2.5-Math-1.5B-Instruct/gguf-q4_0/readme)                         | NLP             | GGUF      | `nexa run mathqwen`                    |
+| [codeqwen](https://www.nexaai.com/Qwen/CodeQwen1.5-7B-Instruct/gguf-q4_0/readme)                        | NLP             | GGUF      | `nexa run codeqwen`                    |
+| [mistral](https://www.nexaai.com/mistralai/Mistral-7B-Instruct-v0.3/gguf-q4_0/readme)                   | NLP             | GGUF/ONNX | `nexa run mistral`                     |
+| [dolphin-mistral](https://www.nexaai.com/CognitiveComputations/dolphin-2.8-mistral-7b/gguf-q4_0/readme) | NLP             | GGUF      | `nexa run dolphin-mistral`             |
+| [codegemma](https://www.nexaai.com/google/codegemma-2b/gguf-q4_0/readme)                                | NLP             | GGUF      | `nexa run codegemma`                   |
+| [codellama](https://www.nexaai.com/meta/CodeLlama-7b-Instruct/gguf-q2_K/readme)                         | NLP             | GGUF      | `nexa run codellama`                   |
+| [deepseek-coder](https://www.nexaai.com/DeepSeek/deepseek-coder-1.3b-instruct/gguf-q4_0/readme)         | NLP             | GGUF      | `nexa run deepseek-coder`              |
+| [phi2](https://www.nexaai.com/microsoft/Phi-2/gguf-q4_0/readme)                                         | NLP             | GGUF      | `nexa run phi2`                        |
+| [phi3](https://www.nexaai.com/microsoft/Phi-3-mini-128k-instruct/gguf-q4_0/readme)                      | NLP             | GGUF/ONNX | `nexa run phi3`                        |
+| [phi3.5](https://nexaai.com/microsoft/Phi-3.5-mini-instruct/gguf-q4_0/readme)                           | NLP             | GGUF      | `nexa run phi3.5`                      |
+| [openelm](https://nexaai.com/apple/OpenELM-3B/gguf-q4_K_M/readme)                                       | NLP             | GGUF      | `nexa run openelm`                     |
+| [AMD-Llama-135m](https://nexaai.com/amd/AMD-Llama-135m/gguf-fp16/readme)                                | NLP             | GGUF      | `nexa run AMD-Llama-135m:fp16`         |
+| [nanollava](https://www.nexaai.com/qnguyen3/nanoLLaVA/gguf-fp16/readme)                                 | Multimodal      | GGUF      | `nexa run nanollava`                   |
+| [llava-phi3](https://www.nexaai.com/xtuner/llava-phi-3-mini/gguf-q4_0/readme)                           | Multimodal      | GGUF      | `nexa run llava-phi3`                  |
+| [llava-llama3](https://www.nexaai.com/xtuner/llava-llama-3-8b-v1.1/gguf-q4_0/readme)                    | Multimodal      | GGUF      | `nexa run llava-llama3`                |
+| [llava1.6-mistral](https://www.nexaai.com/liuhaotian/llava-v1.6-mistral-7b/gguf-q4_0/readme)            | Multimodal      | GGUF      | `nexa run llava1.6-mistral`            |
+| [llava1.6-vicuna](https://www.nexaai.com/liuhaotian/llava-v1.6-vicuna-7b/gguf-q4_0/readme)              | Multimodal      | GGUF      | `nexa run llava1.6-vicuna`             |
+| [stable-diffusion-v1-4](https://www.nexaai.com/runwayml/stable-diffusion-v1-4/gguf-q4_0/readme)         | Computer Vision | GGUF      | `nexa run sd1-4`                       |
+| [stable-diffusion-v1-5](https://www.nexaai.com/runwayml/stable-diffusion-v1-5/gguf-q4_0/readme)         | Computer Vision | GGUF/ONNX | `nexa run sd1-5`                       |
+| [stable-diffusion-v2-1](https://nexaai.com/StabilityAI/stable-diffusion-v2-1/gguf-q4_0/readme)          | Computer Vision | GGUF      | `nexa run sd2-1`                       |
+| [stable-diffusion-3-medium](https://nexaai.com/StabilityAI/stable-diffusion-3-medium/gguf-q4_0/readme)  | Computer Vision | GGUF      | `nexa run sd3`                         |
+| [FLUX.1-schnell](https://nexaai.com/BlackForestLabs/FLUX.1-schnell/gguf-q4_0/readme)                    | Computer Vision | GGUF      | `nexa run flux`                        |
+| [lcm-dreamshaper](https://www.nexaai.com/SimianLuo/lcm-dreamshaper-v7/gguf-fp16/readme)                 | Computer Vision | GGUF/ONNX | `nexa run lcm-dreamshaper`             |
+| [hassaku-lcm](https://nexaai.com/stablediffusionapi/hassaku-hentai-model-v13-LCM/gguf-fp16/readme)      | Computer Vision | GGUF      | `nexa run hassaku-lcm`                 |
+| [anything-lcm](https://www.nexaai.com/Linaqruf/anything-v30-LCM/gguf-fp16/readme)                       | Computer Vision | GGUF      | `nexa run anything-lcm`                |
+| [faster-whisper-tiny](https://www.nexaai.com/Systran/faster-whisper-tiny/bin-cpu-fp16/readme)           | Audio           | BIN       | `nexa run faster-whisper-tiny`         |
+| [faster-whisper-small](https://www.nexaai.com/Systran/faster-whisper-small/bin-cpu-fp16/readme)         | Audio           | BIN       | `nexa run faster-whisper-small`        |
+| [faster-whisper-medium](https://www.nexaai.com/Systran/faster-whisper-medium/bin-cpu-fp16/readme)       | Audio           | BIN       | `nexa run faster-whisper-medium`       |
+| [faster-whisper-base](https://www.nexaai.com/Systran/faster-whisper-base/bin-cpu-fp16/readme)           | Audio           | BIN       | `nexa run faster-whisper-base`         |
+| [faster-whisper-large](https://www.nexaai.com/Systran/faster-whisper-large-v3/bin-cpu-fp16/readme)      | Audio           | BIN       | `nexa run faster-whisper-large`        |
+| [whisper-large-v3-turbo](https://nexaai.com/Systran/faster-whisper-large-v3-turbo/bin-cpu-fp16/readme)  | Audio           | BIN       | `nexa run faster-whisper-large-turbo`  |
+| [whisper-tiny.en](https://nexaai.com/openai/whisper-tiny.en/onnx-cpu-fp32/readme)                       | Audio           | ONNX      | `nexa run whisper-tiny.en`             |
+| [whisper-tiny](https://nexaai.com/openai/whisper-tiny/onnx-cpu-fp32/readme)                             | Audio           | ONNX      | `nexa run whisper-tiny`                |
+| [whisper-small.en](https://nexaai.com/openai/whisper-small.en/onnx-cpu-fp32/readme)                     | Audio           | ONNX      | `nexa run whisper-small.en`            |
+| [whisper-small](https://nexaai.com/openai/whisper-small/onnx-cpu-fp32/readme)                           | Audio           | ONNX      | `nexa run whisper-small`               |
+| [whisper-base.en](https://nexaai.com/openai/whisper-base.en/onnx-cpu-fp32/readme)                       | Audio           | ONNX      | `nexa run whisper-base.en`             |
+| [whisper-base](https://nexaai.com/openai/whisper-base/onnx-cpu-fp32/readme)                             | Audio           | ONNX      | `nexa run whisper-base`                |
+| [mxbai-embed-large-v1](https://nexa.ai/mixedbread-ai/mxbai-embed-large-v1/gguf-fp16/readme)             | Embedding       | GGUF      | `nexa embed mxbai`                     |
+| [nomic-embed-text-v1.5](https://nexa.ai/nomic-ai/nomic-embed-text-v1.5/gguf-fp16/readme)                | Embedding       | GGUF      | `nexa embed nomic`                     |
+| [all-MiniLM-L6-v2](https://nexa.ai/sentence-transformers/all-MiniLM-L6-v2/gguf-fp16/readme)             | Embedding       | GGUF      | `nexa embed all-MiniLM-L6-v2:fp16`     |
+| [all-MiniLM-L12-v2](https://nexa.ai/sentence-transformers/all-MiniLM-L12-v2/gguf-fp16/readme)           | Embedding       | GGUF      | `nexa embed all-MiniLM-L12-v2:fp16`    |
 
 ## CLI Reference
 

{nexaai-0.0.8.7 → nexaai-0.0.8.8}/README.md

@@ -24,10 +24,18 @@
 Nexa SDK is a comprehensive toolkit for supporting **ONNX** and **GGML** models. It supports text generation, image generation, vision-language models (VLM), and speech-to-text (ASR), and text-to-speech (TTS) capabilities. Additionally, it offers an OpenAI-compatible API server with JSON schema mode for function calling and streaming support, and a user-friendly Streamlit UI. Users can run Nexa SDK in any device with Python environment, and GPU acceleration is supported, including CUDA, Metal, and ROCm. An executable version is also available.
 
 ## Latest News 🔥
-* [2024/09] Nexa now has executables for easy installation: [Install Nexa SDK](https://nexaai.com/download-sdk)
+* [2024/10] Support embedding model: `nexa embed <model_path> <prompt>`
+* [2024/10] Support pull and run supported Computer Vision models in GGUF format from HuggingFace: `nexa run -hf <model_id> -mt COMPUTER_VISION`
+* [2024/10] Support VLM in local server.
+* [2024/10] Added option to customize maximum context window for NLP and VLM models.
+* [2024/10] Support running model from user's local path
+* [2024/10] Added LoRA support for NLP models. 
+* [2024/10] Added support for whisper-large-v3-turbo: `nexa run faster-whisper-large-turbo`
+* [2024/10] Added support for AMD-Llama-135m: `nexa run AMD-Llama-135m:fp16`
+* [2024/09] Nexa now has executables for easy installation: [Install Nexa SDK](https://nexaai.com/download-sdk) ✨
 * [2024/09] Added support for Llama 3.2 models: `nexa run llama3.2`
 * [2024/09] Added support for Qwen2.5, Qwen2.5-coder and Qwen2.5-Math models: `nexa run qwen2.5`
-* [2024/09] Now supporting pulling and running GGUF models from Hugging Face: `nexa run -hf <hf model id>`
+* [2024/09] Support pull and run NLP models in GGUF format from HuggingFace: `nexa run -hf <model_id> -mt NLP`
 * [2024/09] Added support for ROCm
 * [2024/09] Added support for Phi-3.5 models: `nexa run phi3.5`
 * [2024/09] Added support for OpenELM models: `nexa run openelm`
@@ -193,60 +201,66 @@ Below is our differentiation from other similar tools:
 ## Supported Models & Model Hub
 Our on-device model hub offers all types of quantized models (text, image, audio, multimodal) with filters for RAM, file size, Tasks, etc. to help you easily explore models with UI. Explore on-device models at [On-device Model Hub](https://model-hub.nexa4ai.com/)
 
-Some supported models:
-| Model                                                                                                   | Type            | Format    | Command                            |
-| ------------------------------------------------------------------------------------------------------- | --------------- | --------- | ---------------------------------- |
-| [octopus-v2](https://www.nexaai.com/NexaAI/Octopus-v2/gguf-q4_0/readme)                                 | NLP             | GGUF      | `nexa run octopus-v2`              |
-| [octopus-v4](https://www.nexaai.com/NexaAI/Octopus-v4/gguf-q4_0/readme)                                 | NLP             | GGUF      | `nexa run octopus-v4`              |
-| [gpt2](https://nexaai.com/openai/gpt2/gguf-q4_0/readme)                                                 | NLP             | GGUF      | `nexa run gpt2`                    |
-| [tinyllama](https://www.nexaai.com/TinyLlama/TinyLlama-1.1B-Chat-v1.0/gguf-fp16/readme)                 | NLP             | GGUF      | `nexa run tinyllama`               |
-| [llama2](https://www.nexaai.com/meta/Llama2-7b-chat/gguf-q4_0/readme)                                   | NLP             | GGUF/ONNX | `nexa run llama2`                  |
-| [llama2-uncensored](https://www.nexaai.com/georgesung/Llama2-7b-chat-uncensored/gguf-q4_0/readme)       | NLP             | GGUF      | `nexa run llama2-uncensored`       |
-| [llama2-function-calling](https://www.nexaai.com/Trelis/Llama2-7b-function-calling/gguf-q4_K_M/readme)  | NLP             | GGUF      | `nexa run llama2-function-calling` |
-| [llama3](https://www.nexaai.com/meta/Llama3-8B-Instruct/gguf-q4_0/readme)                               | NLP             | GGUF/ONNX | `nexa run llama3`                  |
-| [llama3.1](https://www.nexaai.com/meta/Llama3.1-8B-Instruct/gguf-q4_0/readme)                           | NLP             | GGUF/ONNX | `nexa run llama3.1`                |
-| [llama3.2](https://nexaai.com/meta/Llama3.2-3B-Instruct/gguf-q4_0/readme)                               | NLP             | GGUF      | `nexa run llama3.2`                |
-| [llama3-uncensored](https://www.nexaai.com/Orenguteng/Llama3-8B-Lexi-Uncensored/gguf-q4_K_M/readme)     | NLP             | GGUF      | `nexa run llama3-uncensored`       |
-| [gemma](https://www.nexaai.com/google/gemma-1.1-2b-instruct/gguf-q4_0/readme)                           | NLP             | GGUF/ONNX | `nexa run gemma`                   |
-| [gemma2](https://www.nexaai.com/google/gemma-2-2b-instruct/gguf-q4_0/readme)                            | NLP             | GGUF      | `nexa run gemma2`                  |
-| [qwen1.5](https://www.nexaai.com/Qwen/Qwen1.5-7B-Instruct/gguf-q4_0/readme)                             | NLP             | GGUF      | `nexa run qwen1.5`                 |
-| [qwen2](https://www.nexaai.com/Qwen/Qwen2-1.5B-Instruct/gguf-q4_0/readme)                               | NLP             | GGUF/ONNX | `nexa run qwen2`                   |
-| [qwen2.5](https://www.nexaai.com/Qwen/Qwen2.5-1.5B-Instruct/gguf-q4_0/readme)                           | NLP             | GGUF      | `nexa run qwen2.5`                 |
-| [mathqwen](https://nexaai.com/Qwen/Qwen2.5-Math-1.5B-Instruct/gguf-q4_0/readme)                         | NLP             | GGUF      | `nexa run mathqwen`                |
-| [codeqwen](https://www.nexaai.com/Qwen/CodeQwen1.5-7B-Instruct/gguf-q4_0/readme)                        | NLP             | GGUF      | `nexa run codeqwen`                |
-| [mistral](https://www.nexaai.com/mistralai/Mistral-7B-Instruct-v0.3/gguf-q4_0/readme)                   | NLP             | GGUF/ONNX | `nexa run mistral`                 |
-| [dolphin-mistral](https://www.nexaai.com/CognitiveComputations/dolphin-2.8-mistral-7b/gguf-q4_0/readme) | NLP             | GGUF      | `nexa run dolphin-mistral`         |
-| [codegemma](https://www.nexaai.com/google/codegemma-2b/gguf-q4_0/readme)                                | NLP             | GGUF      | `nexa run codegemma`               |
-| [codellama](https://www.nexaai.com/meta/CodeLlama-7b-Instruct/gguf-q2_K/readme)                         | NLP             | GGUF      | `nexa run codellama`               |
-| [deepseek-coder](https://www.nexaai.com/DeepSeek/deepseek-coder-1.3b-instruct/gguf-q4_0/readme)         | NLP             | GGUF      | `nexa run deepseek-coder`          |
-| [phi2](https://www.nexaai.com/microsoft/Phi-2/gguf-q4_0/readme)                                         | NLP             | GGUF      | `nexa run phi2`                    |
-| [phi3](https://www.nexaai.com/microsoft/Phi-3-mini-128k-instruct/gguf-q4_0/readme)                      | NLP             | GGUF/ONNX | `nexa run phi3`                    |
-| [phi3.5](https://nexaai.com/microsoft/Phi-3.5-mini-instruct/gguf-q4_0/readme)                           | NLP             | GGUF      | `nexa run phi3.5`                  |
-| [openelm](https://nexaai.com/apple/OpenELM-3B/gguf-q4_K_M/readme)                                       | NLP             | GGUF      | `nexa run openelm`                 |
-| [nanollava](https://www.nexaai.com/qnguyen3/nanoLLaVA/gguf-fp16/readme)                                 | Multimodal      | GGUF      | `nexa run nanollava`               |
-| [llava-phi3](https://www.nexaai.com/xtuner/llava-phi-3-mini/gguf-q4_0/readme)                           | Multimodal      | GGUF      | `nexa run llava-phi3`              |
-| [llava-llama3](https://www.nexaai.com/xtuner/llava-llama-3-8b-v1.1/gguf-q4_0/readme)                    | Multimodal      | GGUF      | `nexa run llava-llama3`            |
-| [llava1.6-mistral](https://www.nexaai.com/liuhaotian/llava-v1.6-mistral-7b/gguf-q4_0/readme)            | Multimodal      | GGUF      | `nexa run llava1.6-mistral`        |
-| [llava1.6-vicuna](https://www.nexaai.com/liuhaotian/llava-v1.6-vicuna-7b/gguf-q4_0/readme)              | Multimodal      | GGUF      | `nexa run llava1.6-vicuna`         |
-| [stable-diffusion-v1-4](https://www.nexaai.com/runwayml/stable-diffusion-v1-4/gguf-q4_0/readme)         | Computer Vision | GGUF      | `nexa run sd1-4`                   |
-| [stable-diffusion-v1-5](https://www.nexaai.com/runwayml/stable-diffusion-v1-5/gguf-q4_0/readme)         | Computer Vision | GGUF/ONNX | `nexa run sd1-5`                   |
-| [stable-diffusion-v2-1](https://nexaai.com/StabilityAI/stable-diffusion-v2-1/gguf-q4_0/readme)          | Computer Vision | GGUF      | `nexa run sd2-1`                   |
-| [stable-diffusion-3-medium](https://nexaai.com/StabilityAI/stable-diffusion-3-medium/gguf-q4_0/readme)  | Computer Vision | GGUF      | `nexa run sd3`                     |
-| [FLUX.1-schnell](https://nexaai.com/BlackForestLabs/FLUX.1-schnell/gguf-q4_0/readme)                    | Computer Vision | GGUF      | `nexa run flux`                    |
-| [lcm-dreamshaper](https://www.nexaai.com/SimianLuo/lcm-dreamshaper-v7/gguf-fp16/readme)                 | Computer Vision | GGUF/ONNX | `nexa run lcm-dreamshaper`         |
-| [hassaku-lcm](https://nexaai.com/stablediffusionapi/hassaku-hentai-model-v13-LCM/gguf-fp16/readme)      | Computer Vision | GGUF      | `nexa run hassaku-lcm`             |
-| [anything-lcm](https://www.nexaai.com/Linaqruf/anything-v30-LCM/gguf-fp16/readme)                       | Computer Vision | GGUF      | `nexa run anything-lcm`            |
-| [faster-whisper-tiny](https://www.nexaai.com/Systran/faster-whisper-tiny/bin-cpu-fp16/readme)           | Audio           | BIN       | `nexa run faster-whisper-tiny`     |
-| [faster-whisper-small](https://www.nexaai.com/Systran/faster-whisper-small/bin-cpu-fp16/readme)         | Audio           | BIN       | `nexa run faster-whisper-small`    |
-| [faster-whisper-medium](https://www.nexaai.com/Systran/faster-whisper-medium/bin-cpu-fp16/readme)       | Audio           | BIN       | `nexa run faster-whisper-medium`   |
-| [faster-whisper-base](https://www.nexaai.com/Systran/faster-whisper-base/bin-cpu-fp16/readme)           | Audio           | BIN       | `nexa run faster-whisper-base`     |
-| [faster-whisper-large](https://www.nexaai.com/Systran/faster-whisper-large-v3/bin-cpu-fp16/readme)      | Audio           | BIN       | `nexa run faster-whisper-large`    |
-| [whisper-tiny.en](https://nexaai.com/openai/whisper-tiny.en/onnx-cpu-fp32/readme)                       | Audio           | ONNX      | `nexa run whisper-tiny.en`         |
-| [whisper-tiny](https://nexaai.com/openai/whisper-tiny/onnx-cpu-fp32/readme)                             | Audio           | ONNX      | `nexa run whisper-tiny`            |
-| [whisper-small.en](https://nexaai.com/openai/whisper-small.en/onnx-cpu-fp32/readme)                     | Audio           | ONNX      | `nexa run whisper-small.en`        |
-| [whisper-small](https://nexaai.com/openai/whisper-small/onnx-cpu-fp32/readme)                           | Audio           | ONNX      | `nexa run whisper-small`           |
-| [whisper-base.en](https://nexaai.com/openai/whisper-base.en/onnx-cpu-fp32/readme)                       | Audio           | ONNX      | `nexa run whisper-base.en`         |
-| [whisper-base](https://nexaai.com/openai/whisper-base/onnx-cpu-fp32/readme)                             | Audio           | ONNX      | `nexa run whisper-base`            |
+Supported models (full list at [Model Hub](https://nexa.ai/models)):
+| Model                                                                                                   | Type            | Format    | Command                                |
+| ------------------------------------------------------------------------------------------------------- | --------------- | --------- | -------------------------------------- |
+| [octopus-v2](https://www.nexaai.com/NexaAI/Octopus-v2/gguf-q4_0/readme)                                 | NLP             | GGUF      | `nexa run octopus-v2`                  |
+| [octopus-v4](https://www.nexaai.com/NexaAI/Octopus-v4/gguf-q4_0/readme)                                 | NLP             | GGUF      | `nexa run octopus-v4`                  |
+| [gpt2](https://nexaai.com/openai/gpt2/gguf-q4_0/readme)                                                 | NLP             | GGUF      | `nexa run gpt2`                        |
+| [tinyllama](https://www.nexaai.com/TinyLlama/TinyLlama-1.1B-Chat-v1.0/gguf-fp16/readme)                 | NLP             | GGUF      | `nexa run tinyllama`                   |
+| [llama2](https://www.nexaai.com/meta/Llama2-7b-chat/gguf-q4_0/readme)                                   | NLP             | GGUF/ONNX | `nexa run llama2`                      |
+| [llama2-uncensored](https://www.nexaai.com/georgesung/Llama2-7b-chat-uncensored/gguf-q4_0/readme)       | NLP             | GGUF      | `nexa run llama2-uncensored`           |
+| [llama2-function-calling](https://www.nexaai.com/Trelis/Llama2-7b-function-calling/gguf-q4_K_M/readme)  | NLP             | GGUF      | `nexa run llama2-function-calling`     |
+| [llama3](https://www.nexaai.com/meta/Llama3-8B-Instruct/gguf-q4_0/readme)                               | NLP             | GGUF/ONNX | `nexa run llama3`                      |
+| [llama3.1](https://www.nexaai.com/meta/Llama3.1-8B-Instruct/gguf-q4_0/readme)                           | NLP             | GGUF/ONNX | `nexa run llama3.1`                    |
+| [llama3.2](https://nexaai.com/meta/Llama3.2-3B-Instruct/gguf-q4_0/readme)                               | NLP             | GGUF      | `nexa run llama3.2`                    |
+| [llama3-uncensored](https://www.nexaai.com/Orenguteng/Llama3-8B-Lexi-Uncensored/gguf-q4_K_M/readme)     | NLP             | GGUF      | `nexa run llama3-uncensored`           |
+| [gemma](https://www.nexaai.com/google/gemma-1.1-2b-instruct/gguf-q4_0/readme)                           | NLP             | GGUF/ONNX | `nexa run gemma`                       |
+| [gemma2](https://www.nexaai.com/google/gemma-2-2b-instruct/gguf-q4_0/readme)                            | NLP             | GGUF      | `nexa run gemma2`                      |
+| [qwen1.5](https://www.nexaai.com/Qwen/Qwen1.5-7B-Instruct/gguf-q4_0/readme)                             | NLP             | GGUF      | `nexa run qwen1.5`                     |
+| [qwen2](https://www.nexaai.com/Qwen/Qwen2-1.5B-Instruct/gguf-q4_0/readme)                               | NLP             | GGUF/ONNX | `nexa run qwen2`                       |
+| [qwen2.5](https://www.nexaai.com/Qwen/Qwen2.5-1.5B-Instruct/gguf-q4_0/readme)                           | NLP             | GGUF      | `nexa run qwen2.5`                     |
+| [mathqwen](https://nexaai.com/Qwen/Qwen2.5-Math-1.5B-Instruct/gguf-q4_0/readme)                         | NLP             | GGUF      | `nexa run mathqwen`                    |
+| [codeqwen](https://www.nexaai.com/Qwen/CodeQwen1.5-7B-Instruct/gguf-q4_0/readme)                        | NLP             | GGUF      | `nexa run codeqwen`                    |
+| [mistral](https://www.nexaai.com/mistralai/Mistral-7B-Instruct-v0.3/gguf-q4_0/readme)                   | NLP             | GGUF/ONNX | `nexa run mistral`                     |
+| [dolphin-mistral](https://www.nexaai.com/CognitiveComputations/dolphin-2.8-mistral-7b/gguf-q4_0/readme) | NLP             | GGUF      | `nexa run dolphin-mistral`             |
+| [codegemma](https://www.nexaai.com/google/codegemma-2b/gguf-q4_0/readme)                                | NLP             | GGUF      | `nexa run codegemma`                   |
+| [codellama](https://www.nexaai.com/meta/CodeLlama-7b-Instruct/gguf-q2_K/readme)                         | NLP             | GGUF      | `nexa run codellama`                   |
+| [deepseek-coder](https://www.nexaai.com/DeepSeek/deepseek-coder-1.3b-instruct/gguf-q4_0/readme)         | NLP             | GGUF      | `nexa run deepseek-coder`              |
+| [phi2](https://www.nexaai.com/microsoft/Phi-2/gguf-q4_0/readme)                                         | NLP             | GGUF      | `nexa run phi2`                        |
+| [phi3](https://www.nexaai.com/microsoft/Phi-3-mini-128k-instruct/gguf-q4_0/readme)                      | NLP             | GGUF/ONNX | `nexa run phi3`                        |
+| [phi3.5](https://nexaai.com/microsoft/Phi-3.5-mini-instruct/gguf-q4_0/readme)                           | NLP             | GGUF      | `nexa run phi3.5`                      |
+| [openelm](https://nexaai.com/apple/OpenELM-3B/gguf-q4_K_M/readme)                                       | NLP             | GGUF      | `nexa run openelm`                     |
+| [AMD-Llama-135m](https://nexaai.com/amd/AMD-Llama-135m/gguf-fp16/readme)                                | NLP             | GGUF      | `nexa run AMD-Llama-135m:fp16`         |
+| [nanollava](https://www.nexaai.com/qnguyen3/nanoLLaVA/gguf-fp16/readme)                                 | Multimodal      | GGUF      | `nexa run nanollava`                   |
+| [llava-phi3](https://www.nexaai.com/xtuner/llava-phi-3-mini/gguf-q4_0/readme)                           | Multimodal      | GGUF      | `nexa run llava-phi3`                  |
+| [llava-llama3](https://www.nexaai.com/xtuner/llava-llama-3-8b-v1.1/gguf-q4_0/readme)                    | Multimodal      | GGUF      | `nexa run llava-llama3`                |
+| [llava1.6-mistral](https://www.nexaai.com/liuhaotian/llava-v1.6-mistral-7b/gguf-q4_0/readme)            | Multimodal      | GGUF      | `nexa run llava1.6-mistral`            |
+| [llava1.6-vicuna](https://www.nexaai.com/liuhaotian/llava-v1.6-vicuna-7b/gguf-q4_0/readme)              | Multimodal      | GGUF      | `nexa run llava1.6-vicuna`             |
+| [stable-diffusion-v1-4](https://www.nexaai.com/runwayml/stable-diffusion-v1-4/gguf-q4_0/readme)         | Computer Vision | GGUF      | `nexa run sd1-4`                       |
+| [stable-diffusion-v1-5](https://www.nexaai.com/runwayml/stable-diffusion-v1-5/gguf-q4_0/readme)         | Computer Vision | GGUF/ONNX | `nexa run sd1-5`                       |
+| [stable-diffusion-v2-1](https://nexaai.com/StabilityAI/stable-diffusion-v2-1/gguf-q4_0/readme)          | Computer Vision | GGUF      | `nexa run sd2-1`                       |
+| [stable-diffusion-3-medium](https://nexaai.com/StabilityAI/stable-diffusion-3-medium/gguf-q4_0/readme)  | Computer Vision | GGUF      | `nexa run sd3`                         |
+| [FLUX.1-schnell](https://nexaai.com/BlackForestLabs/FLUX.1-schnell/gguf-q4_0/readme)                    | Computer Vision | GGUF      | `nexa run flux`                        |
+| [lcm-dreamshaper](https://www.nexaai.com/SimianLuo/lcm-dreamshaper-v7/gguf-fp16/readme)                 | Computer Vision | GGUF/ONNX | `nexa run lcm-dreamshaper`             |
+| [hassaku-lcm](https://nexaai.com/stablediffusionapi/hassaku-hentai-model-v13-LCM/gguf-fp16/readme)      | Computer Vision | GGUF      | `nexa run hassaku-lcm`                 |
+| [anything-lcm](https://www.nexaai.com/Linaqruf/anything-v30-LCM/gguf-fp16/readme)                       | Computer Vision | GGUF      | `nexa run anything-lcm`                |
+| [faster-whisper-tiny](https://www.nexaai.com/Systran/faster-whisper-tiny/bin-cpu-fp16/readme)           | Audio           | BIN       | `nexa run faster-whisper-tiny`         |
+| [faster-whisper-small](https://www.nexaai.com/Systran/faster-whisper-small/bin-cpu-fp16/readme)         | Audio           | BIN       | `nexa run faster-whisper-small`        |
+| [faster-whisper-medium](https://www.nexaai.com/Systran/faster-whisper-medium/bin-cpu-fp16/readme)       | Audio           | BIN       | `nexa run faster-whisper-medium`       |
+| [faster-whisper-base](https://www.nexaai.com/Systran/faster-whisper-base/bin-cpu-fp16/readme)           | Audio           | BIN       | `nexa run faster-whisper-base`         |
+| [faster-whisper-large](https://www.nexaai.com/Systran/faster-whisper-large-v3/bin-cpu-fp16/readme)      | Audio           | BIN       | `nexa run faster-whisper-large`        |
+| [whisper-large-v3-turbo](https://nexaai.com/Systran/faster-whisper-large-v3-turbo/bin-cpu-fp16/readme)  | Audio           | BIN       | `nexa run faster-whisper-large-turbo`  |
+| [whisper-tiny.en](https://nexaai.com/openai/whisper-tiny.en/onnx-cpu-fp32/readme)                       | Audio           | ONNX      | `nexa run whisper-tiny.en`             |
+| [whisper-tiny](https://nexaai.com/openai/whisper-tiny/onnx-cpu-fp32/readme)                             | Audio           | ONNX      | `nexa run whisper-tiny`                |
+| [whisper-small.en](https://nexaai.com/openai/whisper-small.en/onnx-cpu-fp32/readme)                     | Audio           | ONNX      | `nexa run whisper-small.en`            |
+| [whisper-small](https://nexaai.com/openai/whisper-small/onnx-cpu-fp32/readme)                           | Audio           | ONNX      | `nexa run whisper-small`               |
+| [whisper-base.en](https://nexaai.com/openai/whisper-base.en/onnx-cpu-fp32/readme)                       | Audio           | ONNX      | `nexa run whisper-base.en`             |
+| [whisper-base](https://nexaai.com/openai/whisper-base/onnx-cpu-fp32/readme)                             | Audio           | ONNX      | `nexa run whisper-base`                |
+| [mxbai-embed-large-v1](https://nexa.ai/mixedbread-ai/mxbai-embed-large-v1/gguf-fp16/readme)             | Embedding       | GGUF      | `nexa embed mxbai`                     |
+| [nomic-embed-text-v1.5](https://nexa.ai/nomic-ai/nomic-embed-text-v1.5/gguf-fp16/readme)                | Embedding       | GGUF      | `nexa embed nomic`                     |
+| [all-MiniLM-L6-v2](https://nexa.ai/sentence-transformers/all-MiniLM-L6-v2/gguf-fp16/readme)             | Embedding       | GGUF      | `nexa embed all-MiniLM-L6-v2:fp16`     |
+| [all-MiniLM-L12-v2](https://nexa.ai/sentence-transformers/all-MiniLM-L12-v2/gguf-fp16/readme)           | Embedding       | GGUF      | `nexa embed all-MiniLM-L12-v2:fp16`    |
 
 ## CLI Reference
 

nexaai-0.0.8.8/nexa/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.0.8.8"

{nexaai-0.0.8.7 → nexaai-0.0.8.8}/nexa/cli/entry.py

@@ -208,7 +208,7 @@ def run_onnx_inference(args):
 
 def run_eval_tasks(args):
     try:
-        if 'do-not-answer' in args.tasks:
+        if args.tasks and 'do-not-answer' in args.tasks:
             if not os.getenv('OPENAI_API_KEY'):
                 print("Warning: The 'do-not-answer' task requires an OpenAI API key.")
                 print("Please set your API key in the terminal using the following command:")
@@ -221,9 +221,13 @@ def run_eval_tasks(args):
         
         from nexa.eval.nexa_eval import NexaEval
         evaluator = NexaEval(model_path, args.tasks, args.limit, args.port, args.nctx)
-        evaluator.run_evaluation()
+        if not args.tasks:
+            evaluator.run_perf_eval(args.device, args.new_tokens)
+        else:
+            evaluator.run_evaluation()
     except Exception as e:
-        print(f"Error running evaluation, please run: pip install nexaai[eval]")
+        print("Please run: pip install nexaai[eval]")
+        print(f"Error running evaluation: {e}")
         return
 
 def run_embedding_generation(args):
@@ -257,18 +261,8 @@ def run_embedding_generation(args):
         print("Please refer to our docs to install nexaai package: https://docs.nexaai.com/getting-started/installation")
 
 def main():
-    parser = argparse.ArgumentParser(
-        description="Nexa CLI tool for handling various model operations."
-    )
-
-    parser.add_argument(
-        "-V",
-        "--version",
-        action="version",
-        version=__version__,
-        help="Show the version of the Nexa SDK.",
-    )
-
+    parser = argparse.ArgumentParser(description="Nexa CLI tool for handling various model operations.")
+    parser.add_argument("-V", "--version", action="version", version=__version__, help="Show the version of the Nexa SDK.")
     subparsers = parser.add_subparsers(dest="command", help="sub-command help")
 
     # Run command
@@ -349,7 +343,7 @@ def main():
     server_parser.add_argument("-lp", "--local_path", action="store_true", help="Indicate that the model path provided is the local path, must be used with -mt")
     server_parser.add_argument("-mt", "--model_type", type=str, choices=[e.name for e in ModelType], help="Indicate the model running type, must be used with -lp or -hf")
     server_parser.add_argument("-hf", "--huggingface", action="store_true", help="Load model from Hugging Face Hub, must be used with -mt")
-    server_parser.add_argument("--host", type=str, default="0.0.0.0", help="Host to bind the server to")
+    server_parser.add_argument("--host", type=str, default="localhost", help="Host to bind the server to")
     server_parser.add_argument("--port", type=int, default=8000, help="Port to bind the server to")
     server_parser.add_argument("--reload", action="store_true", help="Enable automatic reloading on code changes")
     server_parser.add_argument("--nctx", type=int, default=2048, help="Maximum context length of the model you're using")
@@ -371,10 +365,18 @@ def main():
     # Benchmark Evaluation
     eval_parser = subparsers.add_parser("eval", help="Evaluate models on specified tasks.")
     eval_parser.add_argument("model_path", type=str, help="Path or identifier for the model in Nexa Model Hub")
-    eval_parser.add_argument("--tasks", type=str, required=True, help="Tasks to evaluate the model on, separated by commas.")
-    eval_parser.add_argument("--limit", type=float, help="Limit the number of examples per task. If <1, limit is a percentage of the total number of examples.", default=None)
-    eval_parser.add_argument("--port", type=int, help="Port to bind the server to", default=8300)
-    eval_parser.add_argument("--nctx", type=int, help="Length of context window", default=4096)
+
+    # General evaluation options
+    general_eval_group = eval_parser.add_argument_group('General evaluation options')
+    general_eval_group.add_argument("--tasks", type=str, help="Tasks to evaluate the model on, separated by commas.")
+    general_eval_group.add_argument("--limit", type=float, help="Limit the number of examples per task. If <1, limit is a percentage of the total number of examples.", default=None)
+    general_eval_group.add_argument("--port", type=int, help="Port to bind the server to", default=8300)
+    general_eval_group.add_argument("--nctx", type=int, help="Length of context window", default=4096)
+
+    # Performance evaluation options
+    perf_eval_group = eval_parser.add_argument_group('Performance evaluation options')
+    perf_eval_group.add_argument("--device", type=str, help="Device to run performance evaluation on, choose from 'cpu', 'cuda', 'mps'", default="cpu")
+    perf_eval_group.add_argument("--new_tokens", type=int, help="Number of new tokens to evaluate", default=100)
 
     # Embed command
     embed_parser = subparsers.add_parser("embed", help="Generate embeddings for a given prompt.")

{nexaai-0.0.8.7 → nexaai-0.0.8.8}/nexa/eval/evaluator.py

@@ -141,7 +141,7 @@ def nexa_evaluate(
             requests[reqtype].append(instance)
 
     # Run LM on inputs, get all outputs
-    for reqtype, reqs in requests.items():
+    for reqtype, reqs in requests.items(): # TODO: probably change to multiprocessing
         eval_logger.info(f"Running {reqtype} requests")
         cloned_reqs = []
         for req in reqs:

{nexaai-0.0.8.7 → nexaai-0.0.8.8}/nexa/eval/nexa_eval.py

@@ -11,14 +11,21 @@ from pathlib import Path
 from contextlib import ExitStack
 from nexa.eval import evaluator
 from nexa.eval.nexa_task.task_manager import TaskManager
-from nexa.eval.utils import make_table, simple_parse_args_string, handle_non_serializable
+from nexa.eval.utils import make_table, handle_non_serializable
 from nexa.gguf.server.nexa_service import run_nexa_ai_service as NexaServer
 from nexa.constants import NEXA_MODEL_EVAL_RESULTS_PATH, NEXA_RUN_MODEL_MAP
+from nexa.eval.nexa_perf import (
+    Benchmark,
+    BenchmarkConfig,
+    InferenceConfig,
+    ProcessConfig,
+    NexaConfig,
+)
 
 logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
 
 class NexaEval:
-    def __init__(self, model_path: str, tasks: str, limit: float = None, port: int = None, nctx: int = None):
+    def __init__(self, model_path: str, tasks: str = None, limit: float = None, port: int = None, nctx: int = None):
         model_path = NEXA_RUN_MODEL_MAP.get(model_path, model_path)
         self.model_path = model_path
         
@@ -30,14 +37,15 @@ class NexaEval:
         self.nctx = nctx if nctx is not None else 4096
         self.initial_port = port if port is not None else 8300
         self.port = self.initial_port
-        self.server_url = f"http://0.0.0.0:{self.port}"
-        output_path = Path(NEXA_MODEL_EVAL_RESULTS_PATH) / self.model_name / self.model_tag / self.tasks.replace(',', '_')
+        self.server_url = f"http://localhost:{self.port}"
+        output_path = Path(NEXA_MODEL_EVAL_RESULTS_PATH) / self.model_name / self.model_tag
+        if self.tasks:
+            output_path = output_path / self.tasks.replace(',', '_')
         self.eval_args = {
             "model": self.model_path,
             "tasks": self.tasks,
             "limit": self.limit,
             "model_args": f"base_url={self.server_url}/v1/completions",
-            "hf_hub_log_args": "",
             "batch_size": 8,
             "output_path": str(output_path),
             "include_path": None,
@@ -48,14 +56,14 @@ class NexaEval:
         while True:
             try:
                 with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
-                    s.bind(('0.0.0.0', self.port))
+                    s.bind(('localhost', self.port))
                     return self.port
             except socket.error:
                 logging.info(f"Port {self.port} is in use, trying {self.port + 1}")
                 self.port += 1
 
     def update_urls(self):
-        self.server_url = f"http://0.0.0.0:{self.port}"
+        self.server_url = f"http://localhost:{self.port}"
         self.eval_args["model_args"] = f"base_url={self.server_url}/v1/completions"
 
     def start_server(self):
@@ -64,7 +72,7 @@ class NexaEval:
         self.server_process = multiprocessing.Process(
             target=NexaServer,
             args=(self.model_path,),
-            kwargs={"host": "0.0.0.0", "port": self.port, "nctx": self.nctx},
+            kwargs={"host": "localhost", "port": self.port, "nctx": self.nctx},
         )
         self.server_process.start()
         logging.info(f"Started server process for model: {self.model_path} on port {self.port} with context window {self.nctx}")
@@ -182,23 +190,62 @@ class NexaEval:
             except Exception as e:
                 logging.error(f"An error occurred during evaluation: {e}")
 
+    def run_perf_eval(self, device: str, new_tokens: int):
+        BENCHMARK_NAME = f"nexa_sdk_{self.model_path}"
+        launcher_config = ProcessConfig()
+        backend_config = NexaConfig(
+            device=device,
+            model=self.model_path,
+            task="text-generation",
+        )
+        scenario_config = InferenceConfig(
+            latency=True,
+            memory=True,  
+            energy=True,
+            input_shapes={
+                "batch_size": 1,        # TODO: make it dynamic, hardcoded to 1 for now
+                "sequence_length": 256,
+                "vocab_size": 32000,
+            },
+            generate_kwargs={
+                "max_new_tokens": new_tokens,
+                "min_new_tokens": new_tokens,
+            },
+        )
+        benchmark_config = BenchmarkConfig(
+            name=BENCHMARK_NAME,
+            launcher=launcher_config,
+            backend=backend_config,
+            scenario=scenario_config,
+        )
+
+        # Launch the benchmark with the specified configuration
+        benchmark_report = Benchmark.launch(benchmark_config)
+        benchmark_report.save_csv(f"benchmark_report_{self.model_path}.csv")
+
+
     def stop_server(self):
         if self.server_process:
             self.server_process.terminate()
             self.server_process.join()
             logging.info("Server process terminated")
 
-def run_eval_inference(model_path: str, tasks: str, limit: float = None, port: int = None, nctx: int = None):
+def run_eval_inference(model_path: str, tasks: str = None, limit: float = None, port: int = None, nctx: int = None, device: str = "cpu", new_tokens: int = 100):
     evaluator = NexaEval(model_path, tasks, limit, port, nctx)
-    evaluator.run_evaluation()
+    if not tasks:
+        evaluator.run_perf_eval(device, new_tokens)
+    else:
+        evaluator.run_evaluation()
 
 if __name__ == "__main__":
     parser = argparse.ArgumentParser(description="Run Nexa Model Evaluation")
     parser.add_argument("model_path", type=str, help="Path or identifier for the model in Nexa Model Hub")
-    parser.add_argument("--tasks", type=str, help="Tasks to evaluate, comma-separated")
+    parser.add_argument("--tasks", type=str, help="Tasks to evaluate, comma-separated", default=None)
     parser.add_argument("--limit", type=float, help="Limit the number of examples per task. If <1, limit is a percentage of the total number of examples.", default=None)
     parser.add_argument("--port", type=int, help="Initial port to bind the server to", default=8300)
     parser.add_argument("--nctx", type=int, help="Length of context window", default=4096)
-    
+    parser.add_argument("--device", type=str, help="Device to run the inference on, choose from 'cpu', 'cuda', 'mps'", default="cpu")
+    parser.add_argument("--new_tokens", type=int, help="Number of new tokens to evaluate", default=100)
+
     args = parser.parse_args()
-    run_eval_inference(args.model_path, args.tasks, args.limit, args.port, args.nctx)
+    run_eval_inference(args.model_path, args.tasks, args.limit, args.port, args.nctx, args.device, args.new_tokens)