vec-inf 0.3.0__tar.gz → 0.3.1__tar.gz

{vec_inf-0.3.0 → vec_inf-0.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: vec-inf
-Version: 0.3.0
+Version: 0.3.1
 Summary: Efficient LLM inference on Slurm clusters using vLLM.
 License: MIT
 Author: Marshall Wang
@@ -22,7 +22,7 @@ Requires-Dist: vllm-nccl-cu12 (>=2.18,<2.19) ; extra == "dev"
 Description-Content-Type: text/markdown
 
 # Vector Inference: Easy inference on Slurm clusters
-This repository provides an easy-to-use solution to run inference servers on [Slurm](https://slurm.schedmd.com/overview.html)-managed computing clusters using [vLLM](https://docs.vllm.ai/en/latest/). **All scripts in this repository runs natively on the Vector Institute cluster environment**. To adapt to other environments, update the config files in the `vec_inf/models` folder and the environment variables in the model launching scripts in `vec_inf` accordingly.  
+This repository provides an easy-to-use solution to run inference servers on [Slurm](https://slurm.schedmd.com/overview.html)-managed computing clusters using [vLLM](https://docs.vllm.ai/en/latest/). **All scripts in this repository runs natively on the Vector Institute cluster environment**. To adapt to other environments, update [`launch_server.sh`](vec-inf/launch_server.sh), [`vllm.slurm`](vec-inf/vllm.slurm), [`multinode_vllm.slurm`](vec-inf/multinode_vllm.slurm) and [`models.csv`](vec-inf/models/models.csv) accordingly.  
 
 ## Installation
 If you are using the Vector cluster environment, and you don't need any customization to the inference server environment, run the following to install package:
@@ -32,15 +32,15 @@ pip install vec-inf
 Otherwise, we recommend using the provided [`Dockerfile`](Dockerfile) to set up your own environment with the package
 
 ## Launch an inference server
-We will use the Llama 3 model as example, to launch an inference server for Llama 3 8B, run:
+We will use the Llama 3.1 model as example, to launch an OpenAI compatible inference server for Meta-Llama-3.1-8B-Instruct, run:
 ```bash
-vec-inf launch llama-3
+vec-inf launch Meta-Llama-3.1-8B-Instruct
 ```
 You should see an output like the following:
 
-<img src="https://github.com/user-attachments/assets/c50646df-0991-4164-ad8f-6eb7e86b67e0" width="350">
+<img width="400" alt="launch_img" src="https://github.com/user-attachments/assets/557eb421-47db-4810-bccd-c49c526b1b43">
 
-There is a default variant for every model family, which is specified in `vec_inf/models/{MODEL_FAMILY_NAME}/README.md`, you can switch to other variants with the `--model-variant` option, and make sure to change the requested resource accordingly. More information about the available options can be found in the [`vec_inf/models`](vec_inf/models) folder. The inference server is compatible with the OpenAI `Completion` and `ChatCompletion` API. 
+The model would be launched using the [default parameters](vec-inf/models/models.csv), you can override these values by providing additional options, use `--help` to see the full list. You can also launch your own customized model as long as the model architecture is [supported by vLLM](https://docs.vllm.ai/en/stable/models/supported_models.html), you'll need to specify all model launching related options to run a successful run. 
 
 You can check the inference server status by providing the Slurm job ID to the `status` command:
 ```bash
@@ -49,18 +49,17 @@ vec-inf status 13014393
 
 You should see an output like the following:
 
-<img src="https://github.com/user-attachments/assets/310086fd-82ea-4bfc-8062-5c8e71c5650c" width="400">
+<img width="400" alt="status_img" src="https://github.com/user-attachments/assets/7385b9ca-9159-4ca9-bae2-7e26d80d9747">
 
 There are 5 possible states:
 
-* **PENDING**: Job submitted to Slurm, but not executed yet.
+* **PENDING**: Job submitted to Slurm, but not executed yet. Job pending reason will be shown.
 * **LAUNCHING**: Job is running but the server is not ready yet.
-* **READY**: Inference server running and ready to take requests.
-* **FAILED**: Inference server in an unhealthy state.
+* **READY**: Inference server running and ready to take requests. 
+* **FAILED**: Inference server in an unhealthy state. Job failed reason will be shown.
 * **SHUTDOWN**: Inference server is shutdown/cancelled.
 
-Note that the base URL is only available when model is in `READY` state. 
-Both `launch` and `status` command supports `--json-mode`, where the output information would be structured as a JSON string.
+Note that the base URL is only available when model is in `READY` state, and if you've changed the Slurm log directory path, you also need to specify it when using the `status` command.
 
 Finally, when you're finished using a model, you can shut it down by providing the Slurm job ID:
 ```bash
@@ -69,15 +68,19 @@ vec-inf shutdown 13014393
 > Shutting down model with Slurm Job ID: 13014393
 ```
 
-Here is a more complicated example that launches a model variant using multiple nodes, say we want to launch Mixtral 8x22B, run
+You call view the full list of available models by running the `list` command:
 ```bash
-vec-inf launch mixtral --model-variant 8x22B-v0.1 --num-nodes 2 --num-gpus 4
+vec-inf list
 ```
+<img width="1200" alt="list_img" src="https://github.com/user-attachments/assets/a4f0d896-989d-43bf-82a2-6a6e5d0d288f">
 
-And for launching a multimodal model, here is an example for launching LLaVa-NEXT Mistral 7B (default variant)
+You can also view the default setup for a specific supported model by providing the model name, for example `Meta-Llama-3.1-70B-Instruct`:
 ```bash
-vec-inf launch llava-v1.6 --is-vlm 
+vec-inf list Meta-Llama-3.1-70B-Instruct
 ```
+<img width="400" alt="list_model_img" src="https://github.com/user-attachments/assets/5dec7a33-ba6b-490d-af47-4cf7341d0b42">
+
+`launch`, `list`, and `status` command supports `--json-mode`, where the command output would be structured as a JSON string.
 
 ## Send inference requests
 Once the inference server is ready, you can start sending in inference requests. We provide example scripts for sending inference requests in [`examples`](examples) folder. Make sure to update the model server URL and the model weights location in the scripts. For example, you can run `python examples/inference/llm/completions.py`, and you should expect to see an output like the following:
@@ -90,5 +93,5 @@ If you want to run inference from your local device, you can open a SSH tunnel t
 ```bash
 ssh -L 8081:172.17.8.29:8081 username@v.vectorinstitute.ai -N
 ```
-The example provided above is for the vector cluster, change the variables accordingly for your environment
+Where the last number in the URL is the GPU number (gpu029 in this case). The example provided above is for the vector cluster, change the variables accordingly for your environment
 

{vec_inf-0.3.0 → vec_inf-0.3.1}/README.md

@@ -1,5 +1,5 @@
 # Vector Inference: Easy inference on Slurm clusters
-This repository provides an easy-to-use solution to run inference servers on [Slurm](https://slurm.schedmd.com/overview.html)-managed computing clusters using [vLLM](https://docs.vllm.ai/en/latest/). **All scripts in this repository runs natively on the Vector Institute cluster environment**. To adapt to other environments, update the config files in the `vec_inf/models` folder and the environment variables in the model launching scripts in `vec_inf` accordingly.  
+This repository provides an easy-to-use solution to run inference servers on [Slurm](https://slurm.schedmd.com/overview.html)-managed computing clusters using [vLLM](https://docs.vllm.ai/en/latest/). **All scripts in this repository runs natively on the Vector Institute cluster environment**. To adapt to other environments, update [`launch_server.sh`](vec-inf/launch_server.sh), [`vllm.slurm`](vec-inf/vllm.slurm), [`multinode_vllm.slurm`](vec-inf/multinode_vllm.slurm) and [`models.csv`](vec-inf/models/models.csv) accordingly.  
 
 ## Installation
 If you are using the Vector cluster environment, and you don't need any customization to the inference server environment, run the following to install package:
@@ -9,15 +9,15 @@ pip install vec-inf
 Otherwise, we recommend using the provided [`Dockerfile`](Dockerfile) to set up your own environment with the package
 
 ## Launch an inference server
-We will use the Llama 3 model as example, to launch an inference server for Llama 3 8B, run:
+We will use the Llama 3.1 model as example, to launch an OpenAI compatible inference server for Meta-Llama-3.1-8B-Instruct, run:
 ```bash
-vec-inf launch llama-3
+vec-inf launch Meta-Llama-3.1-8B-Instruct
 ```
 You should see an output like the following:
 
-<img src="https://github.com/user-attachments/assets/c50646df-0991-4164-ad8f-6eb7e86b67e0" width="350">
+<img width="400" alt="launch_img" src="https://github.com/user-attachments/assets/557eb421-47db-4810-bccd-c49c526b1b43">
 
-There is a default variant for every model family, which is specified in `vec_inf/models/{MODEL_FAMILY_NAME}/README.md`, you can switch to other variants with the `--model-variant` option, and make sure to change the requested resource accordingly. More information about the available options can be found in the [`vec_inf/models`](vec_inf/models) folder. The inference server is compatible with the OpenAI `Completion` and `ChatCompletion` API. 
+The model would be launched using the [default parameters](vec-inf/models/models.csv), you can override these values by providing additional options, use `--help` to see the full list. You can also launch your own customized model as long as the model architecture is [supported by vLLM](https://docs.vllm.ai/en/stable/models/supported_models.html), you'll need to specify all model launching related options to run a successful run. 
 
 You can check the inference server status by providing the Slurm job ID to the `status` command:
 ```bash
@@ -26,18 +26,17 @@ vec-inf status 13014393
 
 You should see an output like the following:
 
-<img src="https://github.com/user-attachments/assets/310086fd-82ea-4bfc-8062-5c8e71c5650c" width="400">
+<img width="400" alt="status_img" src="https://github.com/user-attachments/assets/7385b9ca-9159-4ca9-bae2-7e26d80d9747">
 
 There are 5 possible states:
 
-* **PENDING**: Job submitted to Slurm, but not executed yet.
+* **PENDING**: Job submitted to Slurm, but not executed yet. Job pending reason will be shown.
 * **LAUNCHING**: Job is running but the server is not ready yet.
-* **READY**: Inference server running and ready to take requests.
-* **FAILED**: Inference server in an unhealthy state.
+* **READY**: Inference server running and ready to take requests. 
+* **FAILED**: Inference server in an unhealthy state. Job failed reason will be shown.
 * **SHUTDOWN**: Inference server is shutdown/cancelled.
 
-Note that the base URL is only available when model is in `READY` state. 
-Both `launch` and `status` command supports `--json-mode`, where the output information would be structured as a JSON string.
+Note that the base URL is only available when model is in `READY` state, and if you've changed the Slurm log directory path, you also need to specify it when using the `status` command.
 
 Finally, when you're finished using a model, you can shut it down by providing the Slurm job ID:
 ```bash
@@ -46,15 +45,19 @@ vec-inf shutdown 13014393
 > Shutting down model with Slurm Job ID: 13014393
 ```
 
-Here is a more complicated example that launches a model variant using multiple nodes, say we want to launch Mixtral 8x22B, run
+You call view the full list of available models by running the `list` command:
 ```bash
-vec-inf launch mixtral --model-variant 8x22B-v0.1 --num-nodes 2 --num-gpus 4
+vec-inf list
 ```
+<img width="1200" alt="list_img" src="https://github.com/user-attachments/assets/a4f0d896-989d-43bf-82a2-6a6e5d0d288f">
 
-And for launching a multimodal model, here is an example for launching LLaVa-NEXT Mistral 7B (default variant)
+You can also view the default setup for a specific supported model by providing the model name, for example `Meta-Llama-3.1-70B-Instruct`:
 ```bash
-vec-inf launch llava-v1.6 --is-vlm 
+vec-inf list Meta-Llama-3.1-70B-Instruct
 ```
+<img width="400" alt="list_model_img" src="https://github.com/user-attachments/assets/5dec7a33-ba6b-490d-af47-4cf7341d0b42">
+
+`launch`, `list`, and `status` command supports `--json-mode`, where the command output would be structured as a JSON string.
 
 ## Send inference requests
 Once the inference server is ready, you can start sending in inference requests. We provide example scripts for sending inference requests in [`examples`](examples) folder. Make sure to update the model server URL and the model weights location in the scripts. For example, you can run `python examples/inference/llm/completions.py`, and you should expect to see an output like the following:
@@ -67,4 +70,4 @@ If you want to run inference from your local device, you can open a SSH tunnel t
 ```bash
 ssh -L 8081:172.17.8.29:8081 username@v.vectorinstitute.ai -N
 ```
-The example provided above is for the vector cluster, change the variables accordingly for your environment
+Where the last number in the URL is the GPU number (gpu029 in this case). The example provided above is for the vector cluster, change the variables accordingly for your environment

{vec_inf-0.3.0 → vec_inf-0.3.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "vec-inf"
-version = "0.3.0"
+version = "0.3.1"
 description = "Efficient LLM inference on Slurm clusters using vLLM."
 authors = ["Marshall Wang <marshall.wang@vectorinstitute.ai>"]
 license = "MIT license"

{vec_inf-0.3.0 → vec_inf-0.3.1}/vec_inf/README.md

@@ -1,5 +1,8 @@
 # `vec-inf` Commands
 
 * `launch`: Specify a model family and other optional parameters to launch an OpenAI compatible inference server, `--json-mode` supported. Check [`here`](./models/README.md) for complete list of available options.
+* `list`: List all available model names, `--json-mode` supported.
 * `status`: Check the model status by providing its Slurm job ID, `--json-mode` supported.
-* `shutdown`: Shutdown a model by providing its Slurm job ID. 
+* `shutdown`: Shutdown a model by providing its Slurm job ID. 
+
+Use `--help` to see all available options

{vec_inf-0.3.0 → vec_inf-0.3.1}/vec_inf/cli/_cli.py

@@ -1,7 +1,6 @@
 import os
 
 import click
-import pandas as pd
 from rich.console import Console
 from rich.columns import Columns
 from rich.panel import Panel
@@ -27,12 +26,12 @@ def cli():
 @click.option(
     "--model-family",
     type=str,
-    help='The model family name according to the directories in `models`'
+    help='The model family'
 )
 @click.option(
     "--model-variant",
     type=str,
-    help='The model variant according to the README in `models/model-family`'
+    help='The model variant'
 )
 @click.option(
     "--max-model-len",
@@ -57,12 +56,12 @@ def cli():
 @click.option(
     "--qos",
     type=str,
-    help='Quality of service, default to m3'
+    help='Quality of service, default depends on suggested resource allocation required for the model'
 )
 @click.option(
     "--time",
     type=str,
-    help='Time limit for job, this should comply with QoS, default to 4:00:00'
+    help='Time limit for job, this should comply with QoS, default to max walltime of the chosen QoS'
 )
 @click.option(
     "--data-type",
@@ -77,7 +76,7 @@ def cli():
 @click.option(
     "--log-dir",
     type=str,
-    help='Path to slurm log directory'
+    help='Path to slurm log directory, default to .vec-inf-logs in home directory'
 )
 @click.option(
     "--json-mode",
@@ -150,7 +149,7 @@ def launch(
 @click.option(
     "--log-dir",
     type=str,
-    help='Path to slurm log directory. This is required if it was set when launching the model'
+    help='Path to slurm log directory. This is required if --log-dir was set in model launch'
 )
 @click.option(
     "--json-mode",
@@ -238,16 +237,40 @@ def shutdown(slurm_job_id: int) -> None:
 
 
 @cli.command("list")
+@click.argument(
+    "model-name",
+    required=False)
 @click.option(
     "--json-mode",
     is_flag=True,
     help='Output in JSON string',
 )
-def list(json_mode: bool=False) -> None:
+def list(model_name: str=None, json_mode: bool=False) -> None:
     """
-    List all available models
+    List all available models, or get default setup of a specific model
     """
     models_df = load_models_df()
+
+    if model_name:
+        if model_name not in models_df['model_name'].values:
+            raise ValueError(f"Model name {model_name} not found in available models")
+        
+        excluded_keys = {'venv', 'log_dir', 'pipeline_parallelism'}
+        model_row = models_df.loc[models_df['model_name'] == model_name]
+
+        if json_mode:
+            # click.echo(model_row.to_json(orient='records'))
+            filtered_model_row = model_row.drop(columns=excluded_keys, errors='ignore')
+            click.echo(filtered_model_row.to_json(orient='records'))
+            return
+        table = create_table(key_title="Model Config", value_title="Value")
+        for _, row in model_row.iterrows():
+            for key, value in row.items():
+                if key not in excluded_keys:
+                    table.add_row(key, str(value))
+        CONSOLE.print(table)
+        return
+    
     if json_mode:
         click.echo(models_df['model_name'].to_json(orient='records'))
         return

{vec_inf-0.3.0 → vec_inf-0.3.1}/vec_inf/launch_server.sh

@@ -22,7 +22,7 @@ while [[ "$#" -gt 0 ]]; do
     shift
 done
 
-required_vars=(model_family model_variant partition qos walltime num_nodes num_gpus max_model_len vocab_size data_type virtual_env log_dir pipeline_parallelism)
+required_vars=(model_family model_variant partition qos walltime num_nodes num_gpus max_model_len vocab_size)
 
 for var in "$required_vars[@]"; do
     if [ -z "$!var" ]; then
@@ -40,10 +40,28 @@ export NUM_NODES=$num_nodes
 export NUM_GPUS=$num_gpus
 export VLLM_MAX_MODEL_LEN=$max_model_len
 export VLLM_MAX_LOGPROBS=$vocab_size
-export VLLM_DATA_TYPE=$data_type
-export VENV_BASE=$virtual_env
-export LOG_DIR=$log_dir
-export PIPELINE_PARALLELISM=$pipeline_parallelism
+# For custom models, the following are set to default if not specified
+export VLLM_DATA_TYPE="auto"
+export VENV_BASE="singularity"
+export LOG_DIR="default"
+# Pipeline parallelism is disabled and can only be enabled if specified in models.csv as this is an experimental feature
+export PIPELINE_PARALLELISM="false"
+
+if [ -n "$data_type" ]; then
+    export VLLM_DATA_TYPE=$data_type
+fi
+
+if [ -n "$virtual_env" ]; then
+    export VENV_BASE=$virtual_env
+fi
+
+if [ -n "$log_dir" ]; then
+    export LOG_DIR=$log_dir
+fi
+
+if [ -n "$pipeline_parallelism" ]; then
+    export PIPELINE_PARALLELISM=$pipeline_parallelism
+fi
 
 # ================================= Set default environment variables ======================================
 # Slurm job configuration

vec_inf-0.3.1/vec_inf/models/README.md

@@ -0,0 +1,106 @@
+# Available Models
+More profiling metrics coming soon!
+
+## [Cohere for AI: Command R](https://huggingface.co/collections/CohereForAI/c4ai-command-r-plus-660ec4c34f7a69c50ce7f7b9) 
+
+| Variant | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
+|:----------:|:----------:|:----------:|:----------:|
+|[`c4ai-command-r-plus`](https://huggingface.co/CohereForAI/c4ai-command-r-plus)| 8x a40 (2 nodes, 4 a40/node) | 412 tokens/s | 541 tokens/s |
+
+## [Code Llama](https://huggingface.co/collections/meta-llama/code-llama-family-661da32d0a9d678b6f55b933) 
+
+| Variant | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
+|:----------:|:----------:|:----------:|:----------:|
+| [`CodeLlama-7b-hf`](https://huggingface.co/meta-llama/CodeLlama-7b-hf) | 1x a40 | - tokens/s | - tokens/s |
+| [`CodeLlama-7b-Instruct-hf`](https://huggingface.co/meta-llama/CodeLlama-7b-Instruct-hf) | 1x a40 | - tokens/s | - tokens/s |
+| [`CodeLlama-13b-hf`](https://huggingface.co/meta-llama/CodeLlama-13b-hf) | 1x a40 | - tokens/s | - tokens/s |
+| [`CodeLlama-13b-Instruct-hf`](https://huggingface.co/meta-llama/CodeLlama-13b-Instruct-hf) | 1x a40 | - tokens/s | - tokens/s |
+| [`CodeLlama-34b-hf`](https://huggingface.co/meta-llama/CodeLlama-34b-hf) | 2x a40 | - tokens/s | - tokens/s |
+| [`CodeLlama-34b-Instruct-hf`](https://huggingface.co/meta-llama/CodeLlama-34b-Instruct-hf) | 2x a40 | - tokens/s | - tokens/s |
+| [`CodeLlama-70b-hf`](https://huggingface.co/meta-llama/CodeLlama-70b-hf) | 4x a40 | - tokens/s | - tokens/s |
+| [`CodeLlama-70b-Instruct-hf`](https://huggingface.co/meta-llama/CodeLlama-70b-Instruct-hf) | 4x a40 | - tokens/s | - tokens/s |
+
+## [Databricks: DBRX](https://huggingface.co/collections/databricks/dbrx-6601c0852a0cdd3c59f71962) 
+
+| Variant | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
+|:----------:|:----------:|:----------:|:----------:|
+|[`dbrx-instruct`](https://huggingface.co/databricks/dbrx-instruct)| 8x a40 (2 nodes, 4 a40/node) | 107 tokens/s | 904 tokens/s |
+
+## [Google: Gemma 2](https://huggingface.co/collections/google/gemma-2-release-667d6600fd5220e7b967f315) 
+
+| Variant | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
+|:----------:|:----------:|:----------:|:----------:|
+| [`gemma-2-9b`](https://huggingface.co/google/gemma-2-9b) | 1x a40 | - tokens/s | - tokens/s |
+| [`gemma-2-9b-it`](https://huggingface.co/google/gemma-2-9b-it) | 1x a40 | - tokens/s | - tokens/s |
+| [`gemma-2-27b`](https://huggingface.co/google/gemma-2-27b) | 2x a40 | - tokens/s | - tokens/s |
+| [`gemma-2-27b-it`](https://huggingface.co/google/gemma-2-27b-it) | 2x a40 | - tokens/s | - tokens/s |
+
+## [LLaVa-1.5](https://huggingface.co/collections/llava-hf/llava-15-65f762d5b6941db5c2ba07e0) 
+
+| Variant | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
+|:----------:|:----------:|:----------:|:----------:|
+|[`llava-1.5-7b-hf`](https://huggingface.co/llava-hf/llava-1.5-7b-hf)| 1x a40 | - tokens/s | - tokens/s |
+|[`llava-1.5-13b-hf`](https://huggingface.co/llava-hf/llava-1.5-13b-hf)| 1x a40 | - tokens/s | - tokens/s |
+
+## [LLaVa-NeXT](https://huggingface.co/collections/llava-hf/llava-next-65f75c4afac77fd37dbbe6cf) 
+
+| Variant | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
+|:----------:|:----------:|:----------:|:----------:|
+|[`llava-v1.6-mistral-7b-hf`](https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf)| 1x a40 | - tokens/s | - tokens/s |
+|[`llava-v1.6-34b-hf`](https://huggingface.co/llava-hf/llava-v1.6-34b-hf)| 2x a40 | - tokens/s | - tokens/s |
+
+## [Meta: Llama 2](https://huggingface.co/collections/meta-llama/llama-2-family-661da1f90a9d678b6f55773b) 
+
+| Variant | Suggested resource allocation |
+|:----------:|:----------:|
+| [`Llama-2-7b-hf`](https://huggingface.co/meta-llama/Llama-2-7b-hf) | 1x a40 | 
+| [`Llama-2-7b-chat-hf`](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) | 1x a40 | 
+| [`Llama-2-13b-hf`](https://huggingface.co/meta-llama/Llama-2-13b-hf) | 1x a40 | 
+| [`Llama-2-13b-chat-hf`](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf) | 1x a40 |
+| [`Llama-2-70b-hf`](https://huggingface.co/meta-llama/Llama-2-70b-hf) | 4x a40 |
+| [`Llama-2-70b-chat-hf`](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf) | 4x a40 |
+
+## [Meta: Llama 3](https://huggingface.co/collections/meta-llama/meta-llama-3-66214712577ca38149ebb2b6) 
+
+| Variant | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
+|:----------:|:----------:|:----------:|:----------:|
+| [`Meta-Llama-3-8B`](https://huggingface.co/meta-llama/Meta-Llama-3-8B) | 1x a40 | 222 tokens/s | 1811 tokens/s |
+| [`Meta-Llama-3-8B-Instruct`](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) | 1x a40 | 371 tokens/s | 1990 tokens/s |
+| [`Meta-Llama-3-70B`](https://huggingface.co/meta-llama/Meta-Llama-3-70B) | 4x a40 | 81 tokens/s | 618 tokens/s |
+| [`Meta-Llama-3-70B-Instruct`](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) | 4x a40 | 301 tokens/s | 660 tokens/s |
+
+## [Meta: Llama 3.1](https://huggingface.co/collections/meta-llama/llama-31-669fc079a0c406a149a5738f) 
+
+| Variant | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
+|:----------:|:----------:|:----------:|:----------:|
+| [`Meta-Llama-3.1-8B`](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B) | 1x a40 | - tokens/s | - tokens/s |
+| [`Meta-Llama-3.1-8B-Instruct`](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) | 1x a40 | - tokens/s | - tokens/s |
+| [`Meta-Llama-3.1-70B`](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B) | 4x a40 | - tokens/s | - tokens/s |
+| [`Meta-Llama-3.1-70B-Instruct`](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) | 4x a40 | - tokens/s | - tokens/s |
+| [`Meta-Llama-3.1-405B-Instruct`](https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct) | 32x a40 (8 nodes, 4 a40/node) | - tokens/s | - tokens/s |
+
+## [Mistral AI: Mistral](https://huggingface.co/mistralai) 
+
+| Variant (Mistral) | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
+|:----------:|:----------:|:----------:|:----------:|
+|[`Mistral-7B-v0.1`](https://huggingface.co/mistralai/Mistral-7B-v0.1)| 1x a40 | - tokens/s | - tokens/s|
+|[`Mistral-7B-Instruct-v0.1`](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1)| 1x a40 | - tokens/s | - tokens/s|
+|[`Mistral-7B-Instruct-v0.2`](https://huggingface.co/mistralai/Mistral-7B-v0.2)| 1x a40 | - tokens/s | - tokens/s|
+|[`Mistral-7B-v0.3`](https://huggingface.co/mistralai/Mistral-7B-v0.3)| 1x a40 | - tokens/s | - tokens/s |
+|[`Mistral-7B-Instruct-v0.3`](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3)| 1x a40 | - tokens/s | - tokens/s|
+|[`Mistral-Large-Instruct-2407`](https://huggingface.co/mistralai/Mistral-Large-Instruct-2407)| 4x a40 | - tokens/s | - tokens/s|
+
+## [Mistral AI: Mixtral](https://huggingface.co/mistralai) 
+
+| Variant (Mixtral) | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
+|:----------:|:----------:|:----------:|:----------:|
+|[`Mixtral-8x7B-Instruct-v0.1`](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1)| 4x a40 | 222 tokens/s | 1543 tokens/s |
+|[`Mixtral-8x22B-v0.1`](https://huggingface.co/mistralai/Mixtral-8x22B-v0.1)| 8x a40 (2 nodes, 4 a40/node) | 145 tokens/s | 827 tokens/s|
+|[`Mixtral-8x22B-Instruct-v0.1`](https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1)| 8x a40 (2 nodes, 4 a40/node) | 95 tokens/s | 803 tokens/s|
+
+## [Microsoft: Phi 3](https://huggingface.co/collections/microsoft/phi-3-6626e15e9585a200d2d761e3) 
+
+| Variant | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
+|:----------:|:----------:|:----------:|:----------:|
+| [`Phi-3-medium-128k-instruct`](https://huggingface.co/microsoft/Phi-3-medium-128k-instruct) | 2x a40 | - tokens/s | - tokens/s |
+| [`Phi-3-vision-128k-instruct`](https://huggingface.co/microsoft/Phi-3-vision-128k-instruct) | 2x a40 | - tokens/s | - tokens/s |

vec_inf-0.3.0/vec_inf/models/CodeLlama/README.md

@@ -1,12 +0,0 @@
-# [Code Llama](https://huggingface.co/collections/meta-llama/code-llama-family-661da32d0a9d678b6f55b933) 
-
-| Variant | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
-|:----------:|:----------:|:----------:|:----------:|
-| [`7b-hf`](https://huggingface.co/meta-llama/CodeLlama-7b-hf) | 1x a40 | - tokens/s | - tokens/s |
-| [**`7b-Instruct-hf`**](https://huggingface.co/meta-llama/CodeLlama-7b-Instruct-hf) | 1x a40 | - tokens/s | - tokens/s |
-| [`13b-hf`](https://huggingface.co/meta-llama/CodeLlama-13b-hf) | 1x a40 | - tokens/s | - tokens/s |
-| [`13b-Instruct-hf`](https://huggingface.co/meta-llama/CodeLlama-13b-Instruct-hf) | 1x a40 | - tokens/s | - tokens/s |
-| [`34b-hf`](https://huggingface.co/meta-llama/CodeLlama-34b-hf) | 2x a40 | - tokens/s | - tokens/s |
-| [`34b-Instruct-hf`](https://huggingface.co/meta-llama/CodeLlama-34b-Instruct-hf) | 2x a40 | - tokens/s | - tokens/s |
-| [`70b-hf`](https://huggingface.co/meta-llama/CodeLlama-70b-hf) | 4x a40 | - tokens/s | - tokens/s |
-| [`70b-Instruct-hf`](https://huggingface.co/meta-llama/CodeLlama-70b-Instruct-hf) | 4x a40 | - tokens/s | - tokens/s |

vec_inf-0.3.0/vec_inf/models/CodeLlama/config.sh

@@ -1,5 +0,0 @@
-export MODEL_NAME="CodeLlama"
-export MODEL_VARIANT="7b-Instruct-hf"
-export NUM_NODES=1
-export NUM_GPUS=1
-export VLLM_MAX_LOGPROBS=32000

vec_inf-0.3.0/vec_inf/models/Llama-2/README.md

@@ -1,10 +0,0 @@
-# [Meta: Llama 2](https://huggingface.co/collections/meta-llama/llama-2-family-661da1f90a9d678b6f55773b) 
-
-| Variant | Suggested resource allocation |
-|:----------:|:----------:|
-| [**`7b-hf`**](https://huggingface.co/meta-llama/Llama-2-7b-hf) | 1x a40 | 
-| [`7b-chat-hf`](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) | 1x a40 | 
-| [`13b-hf`](https://huggingface.co/meta-llama/Llama-2-13b-hf) | 1x a40 | 
-| [`13b-chat-hf`](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf) | 1x a40 |
-| [`70b-hf`](https://huggingface.co/meta-llama/Llama-2-70b-hf) | 4x a40 |
-| [`70b-chat-hf`](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf) | 4x a40 |

vec_inf-0.3.0/vec_inf/models/Llama-2/config.sh

@@ -1,5 +0,0 @@
-export MODEL_NAME="Llama-2"
-export MODEL_VARIANT="7b-chat-hf"
-export NUM_NODES=1
-export NUM_GPUS=1
-export VLLM_MAX_LOGPROBS=32000

vec_inf-0.3.0/vec_inf/models/Meta-Llama-3/README.md

@@ -1,8 +0,0 @@
-# [Meta: Llama 3](https://huggingface.co/collections/meta-llama/meta-llama-3-66214712577ca38149ebb2b6) 
-
-| Variant | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
-|:----------:|:----------:|:----------:|:----------:|
-| [`8B`](https://huggingface.co/meta-llama/Meta-Llama-3-8B) | 1x a40 | 222 tokens/s | 1811 tokens/s |
-| [**`8B-Instruct`**](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) | 1x a40 | 371 tokens/s | 1990 tokens/s |
-| [`70B`](https://huggingface.co/meta-llama/Meta-Llama-3-70B) | 4x a40 | 81 tokens/s | 618 tokens/s |
-| [`70B-Instruct`](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) | 4x a40 | 301 tokens/s | 660 tokens/s |

vec_inf-0.3.0/vec_inf/models/Meta-Llama-3/config.sh

@@ -1,5 +0,0 @@
-export MODEL_NAME="Meta-Llama-3"
-export MODEL_VARIANT="8B-Instruct"
-export NUM_NODES=1
-export NUM_GPUS=1
-export VLLM_MAX_LOGPROBS=128256

vec_inf-0.3.0/vec_inf/models/Meta-Llama-3.1/README.md

@@ -1,8 +0,0 @@
-# [Meta: Llama 3.1](https://huggingface.co/collections/meta-llama/llama-31-669fc079a0c406a149a5738f) 
-
-| Variant | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
-|:----------:|:----------:|:----------:|:----------:|
-| [`8B`](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B) | 1x a40 | - tokens/s | - tokens/s |
-| [**`8B-Instruct`**](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) | 1x a40 | - tokens/s | - tokens/s |
-| [`70B`](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B) | 4x a40 | - tokens/s | - tokens/s |
-| [`70B-Instruct`](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) | 4x a40 | - tokens/s | - tokens/s |

vec_inf-0.3.0/vec_inf/models/Meta-Llama-3.1/config.sh

@@ -1,6 +0,0 @@
-export MODEL_NAME="Meta-Llama-3.1"
-export MODEL_VARIANT="8B-Instruct"
-export NUM_NODES=1
-export NUM_GPUS=1
-export VLLM_MAX_LOGPROBS=128256
-export MAX_NUM_SEQS=256

vec_inf-0.3.0/vec_inf/models/Mistral/README.md

@@ -1,10 +0,0 @@
-# [Mistral AI: Mistral](https://huggingface.co/mistralai) 
-* Supported model variants:
-
-| Variant (Mistral) | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
-|:----------:|:----------:|:----------:|:----------:|
-|[`7B-v0.1`](https://huggingface.co/mistralai/Mistral-7B-v0.1)| 1x a40 | - tokens/s | - tokens/s|
-|[`7B-Instruct-v0.1`](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1)| 1x a40 | - tokens/s | - tokens/s|
-|[`7B-Instruct-v0.2`](https://huggingface.co/mistralai/Mistral-7B-v0.2)| 1x a40 | - tokens/s | - tokens/s|
-|[`7B-v0.3`](https://huggingface.co/mistralai/Mistral-7B-v0.3)| 1x a40 | - tokens/s | - tokens/s |
-|[**`7B-Instruct-v0.3`**](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3)| 1x a40 | - tokens/s | - tokens/s|

vec_inf-0.3.0/vec_inf/models/Mistral/config.sh

@@ -1,5 +0,0 @@
-export MODEL_NAME="Mistral"
-export MODEL_VARIANT="7B-Instruct-v0.3"
-export NUM_NODES=1
-export NUM_GPUS=1
-export VLLM_MAX_LOGPROBS=32768

vec_inf-0.3.0/vec_inf/models/Mixtral/README.md

@@ -1,8 +0,0 @@
-# [Mistral AI: Mixtral](https://huggingface.co/mistralai) 
-* Supported model variants:
-
-| Variant (Mixtral) | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
-|:----------:|:----------:|:----------:|:----------:|
-|[**`8x7B-Instruct-v0.1`**](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1)| 4x a40 | 222 tokens/s | 1543 tokens/s |
-|[`8x22B-v0.1`](https://huggingface.co/mistralai/Mixtral-8x22B-v0.1)| 8x a40 (2 nodes, 4 a40/node) | 145 tokens/s | 827 tokens/s|
-|[`8x22B-Instruct-v0.1`](https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1)| 8x a40 (2 nodes, 4 a40/node) | 95 tokens/s | 803 tokens/s|

vec_inf-0.3.0/vec_inf/models/Mixtral/config.sh

@@ -1,5 +0,0 @@
-export MODEL_NAME="Mixtral"
-export MODEL_VARIANT="8x7B-Instruct-v0.1"
-export NUM_NODES=1
-export NUM_GPUS=4
-export VLLM_MAX_LOGPROBS=32000

vec_inf-0.3.0/vec_inf/models/Phi-3/README.md

@@ -1,6 +0,0 @@
-# [Microsoft: Phi 3](https://huggingface.co/collections/microsoft/phi-3-6626e15e9585a200d2d761e3) 
-
-| Variant | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
-|:----------:|:----------:|:----------:|:----------:|
-| [**`medium-128k-instruct`**](https://huggingface.co/microsoft/Phi-3-medium-128k-instruct) | 2x a40 | - tokens/s | - tokens/s |
-| [**`vision-128k-instruct`**](https://huggingface.co/microsoft/Phi-3-vision-128k-instruct) | 2x a40 | - tokens/s | - tokens/s |

vec_inf-0.3.0/vec_inf/models/Phi-3/config.sh

@@ -1,6 +0,0 @@
-export MODEL_NAME="Phi-3"
-export MODEL_VARIANT="medium-128k-instruct"
-export NUM_NODES=1
-export NUM_GPUS=2
-export VLLM_MAX_LOGPROBS=32064
-export MAX_NUM_SEQS=200

vec_inf-0.3.0/vec_inf/models/README.md

@@ -1,31 +0,0 @@
-# Environment Variables
-The following environment variables all have default values that's suitable for the Vector cluster environment. You can use flags to modify certain environment variable values.
-
-* **MODEL_FAMILY**: Directory name of the model family.
-* **SRC_DIR**: Relative path for the `[src](../src/)` folder.
-* **CONFIG_FILE**: Config file containing default values for some environment variables in the **MODEL_FAMILY** diretory.
-* **MODEL_NAME**: Name of model family according to the actual model weights.
-* **MODEL_VARIANT**: Variant of the model, the variants available are listed in respective model folders. Default variant is bolded in the corresponding README.md file.
-* **MODEL_DIR**: Path to model's directory in vector-inference repo.
-* **VLLM_BASE_URL_FILENAME**: The file to store the inference server URL, this file would be generated after launching an inference server, and it would be located in the corresponding model folder with the name `.vllm_{model-name}-{model-variant}_url`.
-* **VENV_BASE**: Location of the virtual environment.
-* **VLLM_MODEL_WEIGHTS**: Location of the model weights.
-* **VLLM_DATA_TYPE**: Model data type.
-* **LD_LIBRARY_PATH**: Include custom locations for dynamically linked library files in a Unix-like operating system. In the script, we tell the dynamic linker to also look at the CUDA and cuDNN directories.
-* **JOB_NAME**: Slurm job name.
-* **NUM_NODES**: Numeber of nodes scheduled. Default to suggested resource allocation.
-* **NUM_GPUS**: Number of GPUs scheduled. Default to suggested resource allocation.
-* **JOB_PARTITION**: Type of compute partition. Default to suggested resource allocation.
-* **QOS**: Quality of Service.
-* **TIME**: Max Walltime.
-
-# Named Arguments
-NOTE: Arguments like `--num-nodes` or `model-variant` might not be available to certain model families because they should fit inside a single node or there is no variant availble in `/model-weights` yet. You can manually add these options in launch scripts if you need, or make a request to download weights for other variants.
-* `--model-variant`: Overrides **MODEL_VARIANT**
-* `--partition`: Overrides **JOB_PARTITION**.
-* `--num-nodes`: Overrides **NUM_NODES**.
-* `--num-gpus`: Overrides **NUM_GPUS**.
-* `--qos`: Overrides **QOS**.
-* `--time`: Overrides **TIME**.
-* `--data-type`: Overrides **VLLM_DATA_TYPE**.
-* `--venv`: Overrides **VENV_BASE**.

vec_inf-0.3.0/vec_inf/models/c4ai-command-r/README.md

@@ -1,5 +0,0 @@
-# [Cohere for AI: Command R](https://huggingface.co/collections/CohereForAI/c4ai-command-r-plus-660ec4c34f7a69c50ce7f7b9) 
-
-| Variant | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
-|:----------:|:----------:|:----------:|:----------:|
-|[**`plus`**](https://huggingface.co/CohereForAI/c4ai-command-r-plus)| 8x a40 (2 nodes, 4 a40/node) | 412 tokens/s | 541 tokens/s |

vec_inf-0.3.0/vec_inf/models/c4ai-command-r/config.sh

@@ -1,5 +0,0 @@
-export MODEL_NAME="c4ai-command-r"
-export MODEL_VARIANT="plus"
-export NUM_NODES=2
-export NUM_GPUS=4
-export VLLM_MAX_LOGPROBS=256000

vec_inf-0.3.0/vec_inf/models/dbrx/README.md

@@ -1,5 +0,0 @@
-# [Databricks: DBRX](https://huggingface.co/collections/databricks/dbrx-6601c0852a0cdd3c59f71962) 
-
-| Variant | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
-|:----------:|:----------:|:----------:|:----------:|
-|[**`dbrx-instruct`**](https://huggingface.co/databricks/dbrx-instruct)| 8x a40 (2 nodes, 4 a40/node) | 107 tokens/s | 904 tokens/s |

vec_inf-0.3.0/vec_inf/models/dbrx/config.sh

@@ -1,5 +0,0 @@
-export MODEL_NAME="dbrx"
-export MODEL_VARIANT="instruct"
-export NUM_NODES=2
-export NUM_GPUS=4
-export VLLM_MAX_LOGPROBS=100352

vec_inf-0.3.0/vec_inf/models/gemma-2/README.md

@@ -1,8 +0,0 @@
-# [Google: Gemma 2](https://huggingface.co/collections/google/gemma-2-release-667d6600fd5220e7b967f315) 
-
-| Variant | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
-|:----------:|:----------:|:----------:|:----------:|
-| [`9b`](https://huggingface.co/google/gemma-2-9b) | 1x a40 | - tokens/s | - tokens/s |
-| [**`9b-it`**](https://huggingface.co/google/gemma-2-9b-it) | 1x a40 | - tokens/s | - tokens/s |
-| [`27b`](https://huggingface.co/google/gemma-2-27b) | 2x a40 | - tokens/s | - tokens/s |
-| [`27b-it`](https://huggingface.co/google/gemma-2-27b-it) | 2x a40 | - tokens/s | - tokens/s |

vec_inf-0.3.0/vec_inf/models/gemma-2/config.sh

@@ -1,6 +0,0 @@
-export MODEL_NAME="gemma-2"
-export MODEL_VARIANT="9b-it"
-export NUM_NODES=1
-export NUM_GPUS=1
-export VLLM_MAX_LOGPROBS=256000
-export VLLM_ATTENTION_BACKEND=FLASHINFER

vec_inf-0.3.0/vec_inf/models/llava-1.5/README.md

@@ -1,7 +0,0 @@
-# [LLaVa-1.5](https://huggingface.co/collections/llava-hf/llava-15-65f762d5b6941db5c2ba07e0) 
-* Supported model variants:
-
-| Variant | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
-|:----------:|:----------:|:----------:|:----------:|
-|[**`7b-hf`**](https://huggingface.co/llava-hf/llava-1.5-7b-hf)| 1x a40 | - tokens/s | - tokens/s |
-|[`13b-hf`](https://huggingface.co/llava-hf/llava-1.5-13b-hf)| 1x a40 | - tokens/s | - tokens/s |

vec_inf-0.3.0/vec_inf/models/llava-1.5/chat_template.jinja

@@ -1,23 +0,0 @@
-{%- if messages[0]['role'] == 'system' -%}
-    {%- set system_message = messages[0]['content'] -%}
-    {%- set messages = messages[1:] -%}
-{%- else -%}
-    {% set system_message = '' -%}
-{%- endif -%}
-
-{{ bos_token + system_message }}
-{%- for message in messages -%}
-    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
-        {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
-    {%- endif -%}
-
-    {%- if message['role'] == 'user' -%}
-        {{ 'USER: ' + message['content'] + '\n' }}
-    {%- elif message['role'] == 'assistant' -%}
-        {{ 'ASSISTANT: ' + message['content'] + eos_token + '\n' }}
-    {%- endif -%}
-{%- endfor -%}
-
-{%- if add_generation_prompt -%}
-    {{ 'ASSISTANT:' }}
-{% endif %}

vec_inf-0.3.0/vec_inf/models/llava-1.5/config.sh

@@ -1,5 +0,0 @@
-export MODEL_NAME="llava-1.5"
-export MODEL_VARIANT="7b-hf"
-export NUM_NODES=1
-export NUM_GPUS=1
-export VLLM_MAX_LOGPROBS=32000

vec_inf-0.3.0/vec_inf/models/llava-v1.6/README.md

@@ -1,7 +0,0 @@
-# [LLaVa-NeXT](https://huggingface.co/collections/llava-hf/llava-next-65f75c4afac77fd37dbbe6cf) 
-* Supported model variants:
-
-| Variant | Suggested resource allocation | Avg prompt throughput | Avg generation throughput |
-|:----------:|:----------:|:----------:|:----------:|
-|[**`mistral-7b-hf`**](https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf)| 1x a40 | - tokens/s | - tokens/s |
-|[`34b-hf`](https://huggingface.co/llava-hf/llava-v1.6-34b-hf)| 2x a40 | - tokens/s | - tokens/s |

vec_inf-0.3.0/vec_inf/models/llava-v1.6/chat_template.jinja

@@ -1,23 +0,0 @@
-{%- if messages[0]['role'] == 'system' -%}
-    {%- set system_message = messages[0]['content'] -%}
-    {%- set messages = messages[1:] -%}
-{%- else -%}
-    {% set system_message = '' -%}
-{%- endif -%}
-
-{{ bos_token + system_message }}
-{%- for message in messages -%}
-    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
-        {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
-    {%- endif -%}
-
-    {%- if message['role'] == 'user' -%}
-        {{ 'USER: ' + message['content'] + '\n' }}
-    {%- elif message['role'] == 'assistant' -%}
-        {{ 'ASSISTANT: ' + message['content'] + eos_token + '\n' }}
-    {%- endif -%}
-{%- endfor -%}
-
-{%- if add_generation_prompt -%}
-    {{ 'ASSISTANT:' }}
-{% endif %}

vec_inf-0.3.0/vec_inf/models/llava-v1.6/config.sh

@@ -1,5 +0,0 @@
-export MODEL_NAME="llava-v1.6"
-export MODEL_VARIANT="mistral-7b-hf"
-export NUM_NODES=1
-export NUM_GPUS=1
-export VLLM_MAX_LOGPROBS=32064