vec-inf 0.5.0__py3-none-any.whl → 0.6.1__py3-none-any.whl

{vec_inf-0.5.0.dist-info → vec_inf-0.6.1.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: vec-inf
-Version: 0.5.0
+Version: 0.6.1
 Summary: Efficient LLM inference on Slurm clusters using vLLM.
 Author-email: Marshall Wang <marshall.wang@vectorinstitute.ai>
 License-Expression: MIT
@@ -25,12 +25,14 @@ Description-Content-Type: text/markdown
 ----------------------------------------------------
 
 [![PyPI](https://img.shields.io/pypi/v/vec-inf)](https://pypi.org/project/vec-inf)
+[![downloads](https://img.shields.io/pypi/dm/vec-inf)](https://pypistats.org/packages/vec-inf)
 [![code checks](https://github.com/VectorInstitute/vector-inference/actions/workflows/code_checks.yml/badge.svg)](https://github.com/VectorInstitute/vector-inference/actions/workflows/code_checks.yml)
-[![docs](https://github.com/VectorInstitute/vector-inference/actions/workflows/docs_deploy.yml/badge.svg)](https://github.com/VectorInstitute/vector-inference/actions/workflows/docs_deploy.yml)
-[![codecov](https://codecov.io/github/VectorInstitute/vector-inference/branch/develop/graph/badge.svg?token=NI88QSIGAC)](https://app.codecov.io/github/VectorInstitute/vector-inference/tree/develop)
+[![docs](https://github.com/VectorInstitute/vector-inference/actions/workflows/docs.yml/badge.svg)](https://github.com/VectorInstitute/vector-inference/actions/workflows/docs.yml)
+[![codecov](https://codecov.io/github/VectorInstitute/vector-inference/branch/main/graph/badge.svg?token=NI88QSIGAC)](https://app.codecov.io/github/VectorInstitute/vector-inference/tree/main)
+[![vLLM](https://img.shields.io/badge/vllm-0.8.5.post1-blue)](https://docs.vllm.ai/en/v0.8.5.post1/index.html)
 ![GitHub License](https://img.shields.io/github/license/VectorInstitute/vector-inference)
 
-This repository provides an easy-to-use solution to run inference servers on [Slurm](https://slurm.schedmd.com/overview.html)-managed computing clusters using [vLLM](https://docs.vllm.ai/en/latest/). **All scripts in this repository runs natively on the Vector Institute cluster environment**. To adapt to other environments, update the environment variables in [`cli/_helper.py`](vec_inf/cli/_helper.py), [`cli/_config.py`](vec_inf/cli/_config.py), [`vllm.slurm`](vec_inf/vllm.slurm), [`multinode_vllm.slurm`](vec_inf/multinode_vllm.slurm) and [`models.yaml`](vec_inf/config/models.yaml) accordingly.
+This repository provides an easy-to-use solution to run inference servers on [Slurm](https://slurm.schedmd.com/overview.html)-managed computing clusters using [vLLM](https://docs.vllm.ai/en/latest/). **All scripts in this repository runs natively on the Vector Institute cluster environment**. To adapt to other environments, update the environment variables in [`vec_inf/client/slurm_vars.py`](vec_inf/client/slurm_vars.py), and the model config for cached model weights in [`vec_inf/config/models.yaml`](vec_inf/config/models.yaml) accordingly.
 
 ## Installation
 If you are using the Vector cluster environment, and you don't need any customization to the inference server environment, run the following to install package:
@@ -38,11 +40,13 @@ If you are using the Vector cluster environment, and you don't need any customiz
 ```bash
 pip install vec-inf
 ```
-Otherwise, we recommend using the provided [`Dockerfile`](Dockerfile) to set up your own environment with the package
+Otherwise, we recommend using the provided [`Dockerfile`](Dockerfile) to set up your own environment with the package. The latest image has `vLLM` version `0.8.5.post1`.
 
 ## Usage
 
-### `launch` command
+Vector Inference provides 2 user interfaces, a CLI and an API
+
+### CLI
 
 The `launch` command allows users to deploy a model as a slurm job. If the job successfully launches, a URL endpoint is exposed for the user to send requests for inference.
 
@@ -53,18 +57,26 @@ vec-inf launch Meta-Llama-3.1-8B-Instruct
 ```
 You should see an output like the following:
 
-<img width="600" alt="launch_img" src="https://github.com/user-attachments/assets/883e6a5b-8016-4837-8fdf-39097dfb18bf">
+<img width="600" alt="launch_image" src="https://github.com/user-attachments/assets/a72a99fd-4bf2-408e-8850-359761d96c4f">
 
 
 #### Overrides
 
-Models that are already supported by `vec-inf` would be launched using the cached configuration or [default configuration](vec_inf/config/models.yaml). You can override these values by providing additional parameters. Use `vec-inf launch --help` to see the full list of parameters that can be
+Models that are already supported by `vec-inf` would be launched using the cached configuration (set in [slurm_vars.py](vec_inf/client/slurm_vars.py)) or [default configuration](vec_inf/config/models.yaml). You can override these values by providing additional parameters. Use `vec-inf launch --help` to see the full list of parameters that can be
 overriden. For example, if `qos` is to be overriden:
 
 ```bash
 vec-inf launch Meta-Llama-3.1-8B-Instruct --qos <new_qos>
 ```
 
+To overwrite default vLLM engine arguments, you can specify the engine arguments in a comma separated string:
+
+```bash
+vec-inf launch Meta-Llama-3.1-8B-Instruct --vllm-args '--max-model-len=65536,--compilation-config=3'
+```
+
+For the full list of vLLM engine arguments, you can find them [here](https://docs.vllm.ai/en/stable/serving/engine_args.html), make sure you select the correct vLLM version.
+
 #### Custom models
 
 You can also launch your own custom model as long as the model architecture is [supported by vLLM](https://docs.vllm.ai/en/stable/models/supported_models.html), and make sure to follow the instructions below:
@@ -89,14 +101,14 @@ models:
     gpus_per_node: 1
     num_nodes: 1
     vocab_size: 152064
-    max_model_len: 1010000
-    max_num_seqs: 256
-    pipeline_parallelism: true
-    enforce_eager: false
     qos: m2
     time: 08:00:00
     partition: a40
     model_weights_parent_dir: /h/<username>/model-weights
+    vllm_args:
+      --max-model-len: 1010000
+      --max-num-seqs: 256
+      --compilation-config: 3
 ```
 
 You would then set the `VEC_INF_CONFIG` path using:
@@ -105,68 +117,44 @@ You would then set the `VEC_INF_CONFIG` path using:
 export VEC_INF_CONFIG=/h/<username>/my-model-config.yaml
 ```
 
-Note that there are other parameters that can also be added to the config but not shown in this example, such as `data_type` and `log_dir`.
+**NOTE**
+* There are other parameters that can also be added to the config but not shown in this example, check the [`ModelConfig`](vec_inf/client/config.py) for details.
+* Check [vLLM Engine Arguments](https://docs.vllm.ai/en/stable/serving/engine_args.html) for the full list of available vLLM engine arguments, the default parallel size for any parallelization is default to 1, so none of the sizes were set specifically in this example
+* For GPU partitions with non-Ampere architectures, e.g. `rtx6000`, `t4v2`, BF16 isn't supported. For models that have BF16 as the default type, when using a non-Ampere GPU, use FP16 instead, i.e. `--dtype: float16`
+* Setting `--compilation-config` to `3` currently breaks multi-node model launches, so we don't set them for models that require multiple nodes of GPUs.
 
-### `status` command
-You can check the inference server status by providing the Slurm job ID to the `status` command:
-```bash
-vec-inf status 15373800
-```
-
-If the server is pending for resources, you should see an output like this:
-
-<img width="400" alt="status_pending_img" src="https://github.com/user-attachments/assets/b659c302-eae1-4560-b7a9-14eb3a822a2f">
+#### Other commands
 
-When the server is ready, you should see an output like this:
+* `status`: Check the model status by providing its Slurm job ID, `--json-mode` supported.
+* `metrics`: Streams performance metrics to the console.
+* `shutdown`: Shutdown a model by providing its Slurm job ID.
+* `list`: List all available model names, or view the default/cached configuration of a specific model, `--json-mode` supported.
 
-<img width="400" alt="status_ready_img" src="https://github.com/user-attachments/assets/672986c2-736c-41ce-ac7c-1fb585cdcb0d">
+For more details on the usage of these commands, refer to the [User Guide](https://vectorinstitute.github.io/vector-inference/user_guide/)
 
-There are 5 possible states:
+### API
 
-* **PENDING**: Job submitted to Slurm, but not executed yet. Job pending reason will be shown.
-* **LAUNCHING**: Job is running but the server is not ready yet.
-* **READY**: Inference server running and ready to take requests.
-* **FAILED**: Inference server in an unhealthy state. Job failed reason will be shown.
-* **SHUTDOWN**: Inference server is shutdown/cancelled.
+Example:
 
-Note that the base URL is only available when model is in `READY` state, and if you've changed the Slurm log directory path, you also need to specify it when using the `status` command.
-
-### `metrics` command
-Once your server is ready, you can check performance metrics by providing the Slurm job ID to the `metrics` command:
-```bash
-vec-inf metrics 15373800
+```python
+>>> from vec_inf.api import VecInfClient
+>>> client = VecInfClient()
+>>> response = client.launch_model("Meta-Llama-3.1-8B-Instruct")
+>>> job_id = response.slurm_job_id
+>>> status = client.get_status(job_id)
+>>> if status.status == ModelStatus.READY:
+...     print(f"Model is ready at {status.base_url}")
+>>> client.shutdown_model(job_id)
 ```
 
-And you will see the performance metrics streamed to your console, note that the metrics are updated with a 2-second interval.
+For details on the usage of the API, refer to the [API Reference](https://vectorinstitute.github.io/vector-inference/api/)
 
-<img width="400" alt="metrics_img" src="https://github.com/user-attachments/assets/3ee143d0-1a71-4944-bbd7-4c3299bf0339">
+## Check Job Configuration
 
-### `shutdown` command
-Finally, when you're finished using a model, you can shut it down by providing the Slurm job ID:
-```bash
-vec-inf shutdown 15373800
-
-> Shutting down model with Slurm Job ID: 15373800
-```
-
-### `list` command
-You call view the full list of available models by running the `list` command:
-```bash
-vec-inf list
-```
-<img width="940" alt="list_img" src="https://github.com/user-attachments/assets/8cf901c4-404c-4398-a52f-0486f00747a3">
-
-NOTE: The above screenshot does not represent the full list of models supported.
-
-You can also view the default setup for a specific supported model by providing the model name, for example `Meta-Llama-3.1-70B-Instruct`:
-```bash
-vec-inf list Meta-Llama-3.1-70B-Instruct
-```
-<img width="500" alt="list_model_img" src="https://github.com/user-attachments/assets/34e53937-2d86-443e-85f6-34e408653ddb">
-
-`launch`, `list`, and `status` command supports `--json-mode`, where the command output would be structured as a JSON string.
+With every model launch, a Slurm script will be generated dynamically based on the job and model configuration. Once the Slurm job is queued, the generated Slurm script will be moved to the log directory for reproducibility, located at `$log_dir/$model_family/$model_name.$slurm_job_id/$model_name.$slurm_job_id.slurm`. In the same directory you can also find a JSON file with the same name that captures the launch configuration, and will have an entry of server URL once the server is ready.
 
 ## Send inference requests
+
 Once the inference server is ready, you can start sending in inference requests. We provide example scripts for sending inference requests in [`examples`](examples) folder. Make sure to update the model server URL and the model weights location in the scripts. For example, you can run `python examples/inference/llm/chat_completions.py`, and you should expect to see an output like the following:
 
 ```json
@@ -199,8 +187,9 @@ Once the inference server is ready, you can start sending in inference requests.
     },
     "prompt_logprobs":null
 }
+
 ```
-**NOTE**: For multimodal models, currently only `ChatCompletion` is available, and only one image can be provided for each prompt.
+**NOTE**: Certain models don't adhere to OpenAI's chat template, e.g. Mistral family. For these models, you can either change your prompt to follow the model's default chat template or provide your own chat template via `--chat-template: TEMPLATE_PATH`.
 
 ## SSH tunnel from your local device
 If you want to run inference from your local device, you can open a SSH tunnel to your cluster environment like the following:

vec_inf-0.6.1.dist-info/RECORD

@@ -0,0 +1,25 @@
+vec_inf/README.md,sha256=3ocJHfV3kRftXFUCdHw3B-p4QQlXuNqkHnjPPNkCgfM,543
+vec_inf/__init__.py,sha256=bHwSIz9lebYuxIemni-lP0h3gwJHVbJnwExQKGJWw_Q,23
+vec_inf/find_port.sh,sha256=bGQ6LYSFVSsfDIGatrSg5YvddbZfaPL0R-Bjo4KYD6I,1088
+vec_inf/cli/__init__.py,sha256=5XIvGQCOnaGl73XMkwetjC-Ul3xuXGrWDXdYJ3aUzvU,27
+vec_inf/cli/_cli.py,sha256=pqZeQr5WxAsV7KSYcUnx_mRL7RnHWk1zf9CcW_ct5uI,10663
+vec_inf/cli/_helper.py,sha256=i1QvJeIT3z7me6bv2Vot5c3NY555Dgo3q8iRlxhOlZ4,13047
+vec_inf/cli/_utils.py,sha256=23vSbmvNOWY1-W1aOAwYqNDkDDmx-5UVlCiXAtxUZ8A,1057
+vec_inf/cli/_vars.py,sha256=V6DrJs_BuUa4yNcbBSSnMwpcyXwEBsizy3D0ubIg2fA,777
+vec_inf/client/__init__.py,sha256=OLlUJ4kL1R-Kh-nXNbvKlAZ3mtHcnozHprVufkVCNWk,739
+vec_inf/client/_client_vars.py,sha256=KG-xImVIzJH3aj5nMUzT9w9LpH-7YGrOew6N77Fj0Js,7638
+vec_inf/client/_exceptions.py,sha256=94Nx_5k1SriJNXzbdnwyXFZolyMutydU08Gsikawzzo,749
+vec_inf/client/_helper.py,sha256=DcEFogbrSb4A8Kc2zixNZNL4nt4iswPk2n5blZgwEWQ,22338
+vec_inf/client/_slurm_script_generator.py,sha256=XYCsadCLDEu9KrrjrNCNgoc0ITmjys9u7yWR9PkFAos,6376
+vec_inf/client/_utils.py,sha256=1dB2O1neEhZNk6MJbBybLQm42vsmEevA2TI0F_kGi0o,8796
+vec_inf/client/api.py,sha256=TYn4lP5Ene8MEuXWYo6ZbGYw9aPnaMlT32SH7jLCifM,9605
+vec_inf/client/config.py,sha256=lPVHwiaGZjKd5M9G7vcsk3DMausFP_telq3JQngBkH8,5080
+vec_inf/client/models.py,sha256=qjocUa5egJTVeVF3962kYOecs1dTaEb2e6TswkYFXM0,6141
+vec_inf/client/slurm_vars.py,sha256=lroK41L4gEVVZNxxE3bEpbKsdMwnH79-7iCKd4zWEa4,1069
+vec_inf/config/README.md,sha256=OlgnD_Ojei_xLkNyS7dGvYMFUzQFqjVRVw0V-QMk_3g,17863
+vec_inf/config/models.yaml,sha256=xImSOjG9yL6LqqYkSLL7_wBZhqKM10-eFaQJ82gP4ig,29420
+vec_inf-0.6.1.dist-info/METADATA,sha256=0YHT8rhEZINfmMF1hQBqU0HBpRbwX-1IeqY_Mla4g28,10682
+vec_inf-0.6.1.dist-info/WHEEL,sha256=qtCwoSJWgHk21S1Kb4ihdzI2rlJ1ZKaIurTj_ngOhyQ,87
+vec_inf-0.6.1.dist-info/entry_points.txt,sha256=uNRXjCuJSR2nveEqD3IeMznI9oVI9YLZh5a24cZg6B0,49
+vec_inf-0.6.1.dist-info/licenses/LICENSE,sha256=mq8zeqpvVSF1EsxmydeXcokt8XnEIfSofYn66S2-cJI,1073
+vec_inf-0.6.1.dist-info/RECORD,,

vec_inf/cli/_config.py

@@ -1,87 +0,0 @@
-"""Model configuration."""
-
-from pathlib import Path
-from typing import Optional, Union
-
-from pydantic import BaseModel, ConfigDict, Field
-from typing_extensions import Literal
-
-
-QOS = Literal[
-    "normal",
-    "m",
-    "m2",
-    "m3",
-    "m4",
-    "m5",
-    "long",
-    "deadline",
-    "high",
-    "scavenger",
-    "llm",
-    "a100",
-]
-
-PARTITION = Literal["a40", "a100", "t4v1", "t4v2", "rtx6000"]
-
-DATA_TYPE = Literal["auto", "float16", "bfloat16", "float32"]
-
-
-class ModelConfig(BaseModel):
-    """Pydantic model for validating and managing model deployment configurations."""
-
-    model_name: str = Field(..., min_length=3, pattern=r"^[a-zA-Z0-9\-_\.]+$")
-    model_family: str = Field(..., min_length=2)
-    model_variant: Optional[str] = Field(
-        default=None, description="Specific variant/version of the model family"
-    )
-    model_type: Literal["LLM", "VLM", "Text_Embedding", "Reward_Modeling"] = Field(
-        ..., description="Type of model architecture"
-    )
-    gpus_per_node: int = Field(..., gt=0, le=8, description="GPUs per node")
-    num_nodes: int = Field(..., gt=0, le=16, description="Number of nodes")
-    vocab_size: int = Field(..., gt=0, le=1_000_000)
-    max_model_len: int = Field(
-        ..., gt=0, le=1_010_000, description="Maximum context length supported"
-    )
-    max_num_seqs: int = Field(
-        default=256, gt=0, le=1024, description="Maximum concurrent request sequences"
-    )
-    compilation_config: int = Field(
-        default=0,
-        gt=-1,
-        le=4,
-        description="torch.compile optimization level",
-    )
-    gpu_memory_utilization: float = Field(
-        default=0.9, gt=0.0, le=1.0, description="GPU memory utilization"
-    )
-    pipeline_parallelism: bool = Field(
-        default=True, description="Enable pipeline parallelism"
-    )
-    enforce_eager: bool = Field(default=False, description="Force eager mode execution")
-    qos: Union[QOS, str] = Field(default="m2", description="Quality of Service tier")
-    time: str = Field(
-        default="08:00:00",
-        pattern=r"^\d{2}:\d{2}:\d{2}$",
-        description="HH:MM:SS time limit",
-    )
-    partition: Union[PARTITION, str] = Field(
-        default="a40", description="GPU partition type"
-    )
-    data_type: Union[DATA_TYPE, str] = Field(
-        default="auto", description="Model precision format"
-    )
-    venv: str = Field(
-        default="singularity", description="Virtual environment/container system"
-    )
-    log_dir: Path = Field(
-        default=Path("~/.vec-inf-logs").expanduser(), description="Log directory path"
-    )
-    model_weights_parent_dir: Path = Field(
-        default=Path("/model-weights"), description="Base directory for model weights"
-    )
-
-    model_config = ConfigDict(
-        extra="forbid", str_strip_whitespace=True, validate_default=True, frozen=True
-    )

vec_inf/multinode_vllm.slurm

@@ -1,154 +0,0 @@
-#!/bin/bash
-#SBATCH --cpus-per-task=16
-#SBATCH --mem=64G
-#SBATCH --exclusive
-#SBATCH --tasks-per-node=1
-
-source ${SRC_DIR}/find_port.sh
-
-if [ "$VENV_BASE" = "singularity" ]; then
-    export SINGULARITY_IMAGE=/model-weights/vec-inf-shared/vector-inference_latest.sif
-    export VLLM_NCCL_SO_PATH=/vec-inf/nccl/libnccl.so.2.18.1
-    module load singularity-ce/3.8.2
-    singularity exec $SINGULARITY_IMAGE ray stop
-fi
-
-# Getting the node names
-nodes=$(scontrol show hostnames "$SLURM_JOB_NODELIST")
-nodes_array=($nodes)
-
-head_node=${nodes_array[0]}
-head_node_ip=$(srun --nodes=1 --ntasks=1 -w "$head_node" hostname --ip-address)
-
-# Find port for head node
-head_node_port=$(find_available_port $head_node_ip 8080 65535)
-
-# Starting the Ray head node
-ip_head=$head_node_ip:$head_node_port
-export ip_head
-echo "IP Head: $ip_head"
-
-echo "Starting HEAD at $head_node"
-if [ "$VENV_BASE" = "singularity" ]; then
-    srun --nodes=1 --ntasks=1 -w "$head_node" \
-        singularity exec --nv --bind ${MODEL_WEIGHTS}:${MODEL_WEIGHTS} $SINGULARITY_IMAGE \
-        ray start --head --node-ip-address="$head_node_ip" --port=$head_node_port \
-        --num-cpus "${SLURM_CPUS_PER_TASK}" --num-gpus "${SLURM_GPUS_PER_NODE}" --block &
-else
-    srun --nodes=1 --ntasks=1 -w "$head_node" \
-        ray start --head --node-ip-address="$head_node_ip" --port=$head_node_port \
-        --num-cpus "${SLURM_CPUS_PER_TASK}" --num-gpus "${SLURM_GPUS_PER_NODE}" --block &
-fi
-
-# Starting the Ray worker nodes
-# Optional, though may be useful in certain versions of Ray < 1.0.
-sleep 10
-
-# number of nodes other than the head node
-worker_num=$((SLURM_JOB_NUM_NODES - 1))
-
-for ((i = 1; i <= worker_num; i++)); do
-    node_i=${nodes_array[$i]}
-    echo "Starting WORKER $i at $node_i"
-    if [ "$VENV_BASE" = "singularity" ]; then
-        srun --nodes=1 --ntasks=1 -w "$node_i" \
-            singularity exec --nv --bind ${MODEL_WEIGHTS}:${MODEL_WEIGHTS} $SINGULARITY_IMAGE \
-            ray start --address "$ip_head" \
-            --num-cpus "${SLURM_CPUS_PER_TASK}" --num-gpus "${SLURM_GPUS_PER_NODE}" --block &
-    else
-        srun --nodes=1 --ntasks=1 -w "$node_i" \
-            ray start --address "$ip_head" \
-            --num-cpus "${SLURM_CPUS_PER_TASK}" --num-gpus "${SLURM_GPUS_PER_NODE}" --block &
-    fi
-
-    sleep 5
-done
-
-
-vllm_port_number=$(find_available_port $head_node_ip 8080 65535)
-
-SERVER_ADDR="http://${head_node_ip}:${vllm_port_number}/v1"
-echo "Server address: $SERVER_ADDR"
-
-jq --arg server_addr "$SERVER_ADDR" \
-  '. + {"server_address": $server_addr}' \
-  "$LOG_DIR/$MODEL_NAME.$SLURM_JOB_ID/$MODEL_NAME.$SLURM_JOB_ID.json" > temp.json \
-  && mv temp.json "$LOG_DIR/$MODEL_NAME.$SLURM_JOB_ID/$MODEL_NAME.$SLURM_JOB_ID.json" \
-  && rm temp.json
-
-if [ "$PIPELINE_PARALLELISM" = "True" ]; then
-    export PIPELINE_PARALLEL_SIZE=$SLURM_JOB_NUM_NODES
-    export TENSOR_PARALLEL_SIZE=$SLURM_GPUS_PER_NODE
-else
-    export PIPELINE_PARALLEL_SIZE=1
-    export TENSOR_PARALLEL_SIZE=$((SLURM_JOB_NUM_NODES*SLURM_GPUS_PER_NODE))
-fi
-
-if [ "$ENFORCE_EAGER" = "True" ]; then
-    export ENFORCE_EAGER="--enforce-eager"
-else
-    export ENFORCE_EAGER=""
-fi
-
-if [ "$ENABLE_PREFIX_CACHING" = "True" ]; then
-    export ENABLE_PREFIX_CACHING="--enable-prefix-caching"
-else
-    export ENABLE_PREFIX_CACHING=""
-fi
-
-if [ "$ENABLE_CHUNKED_PREFILL" = "True" ]; then
-    export ENABLE_CHUNKED_PREFILL="--enable-chunked-prefill"
-else
-    export ENABLE_CHUNKED_PREFILL=""
-fi
-
-if [ -z "$MAX_NUM_BATCHED_TOKENS" ]; then
-    export MAX_NUM_BATCHED_TOKENS=""
-else
-    export MAX_NUM_BATCHED_TOKENS="--max-num-batched-tokens=$MAX_NUM_BATCHED_TOKENS"
-fi
-
-# Activate vllm venv
-if [ "$VENV_BASE" = "singularity" ]; then
-    singularity exec --nv --bind ${MODEL_WEIGHTS}:${MODEL_WEIGHTS} $SINGULARITY_IMAGE \
-    python3.10 -m vllm.entrypoints.openai.api_server \
-    --model ${MODEL_WEIGHTS} \
-    --served-model-name ${MODEL_NAME} \
-    --host "0.0.0.0" \
-    --port ${vllm_port_number} \
-    --pipeline-parallel-size ${PIPELINE_PARALLEL_SIZE} \
-    --tensor-parallel-size ${TENSOR_PARALLEL_SIZE} \
-    --dtype ${DATA_TYPE} \
-    --trust-remote-code \
-    --max-logprobs ${MAX_LOGPROBS} \
-    --max-model-len ${MAX_MODEL_LEN} \
-    --max-num-seqs ${MAX_NUM_SEQS} \
-    --gpu-memory-utilization ${GPU_MEMORY_UTILIZATION} \
-    --compilation-config ${COMPILATION_CONFIG} \
-    --task ${TASK} \
-    ${MAX_NUM_BATCHED_TOKENS} \
-    ${ENABLE_PREFIX_CACHING} \
-    ${ENABLE_CHUNKED_PREFILL} \
-    ${ENFORCE_EAGER}
-else
-    source ${VENV_BASE}/bin/activate
-    python3 -m vllm.entrypoints.openai.api_server \
-    --model ${MODEL_WEIGHTS} \
-    --served-model-name ${MODEL_NAME} \
-    --host "0.0.0.0" \
-    --port ${vllm_port_number} \
-    --pipeline-parallel-size ${PIPELINE_PARALLEL_SIZE} \
-    --tensor-parallel-size ${TENSOR_PARALLEL_SIZE} \
-    --dtype ${DATA_TYPE} \
-    --trust-remote-code \
-    --max-logprobs ${MAX_LOGPROBS} \
-    --max-model-len ${MAX_MODEL_LEN} \
-    --max-num-seqs ${MAX_NUM_SEQS} \
-    --gpu-memory-utilization ${GPU_MEMORY_UTILIZATION} \
-    --compilation-config ${COMPILATION_CONFIG} \
-    --task ${TASK} \
-    ${MAX_NUM_BATCHED_TOKENS} \
-    ${ENABLE_PREFIX_CACHING} \
-    ${ENABLE_CHUNKED_PREFILL} \
-    ${ENFORCE_EAGER}
-fi

vec_inf/vllm.slurm

@@ -1,90 +0,0 @@
-#!/bin/bash
-#SBATCH --cpus-per-task=16
-#SBATCH --mem=64G
-
-source ${SRC_DIR}/find_port.sh
-
-# Write server url to file
-hostname=${SLURMD_NODENAME}
-vllm_port_number=$(find_available_port $hostname 8080 65535)
-
-SERVER_ADDR="http://${hostname}:${vllm_port_number}/v1"
-echo "Server address: $SERVER_ADDR"
-
-jq --arg server_addr "$SERVER_ADDR" \
-  '. + {"server_address": $server_addr}' \
-  "$LOG_DIR/$MODEL_NAME.$SLURM_JOB_ID/$MODEL_NAME.$SLURM_JOB_ID.json" > temp.json \
-  && mv temp.json "$LOG_DIR/$MODEL_NAME.$SLURM_JOB_ID/$MODEL_NAME.$SLURM_JOB_ID.json" \
-  && rm temp.json
-
-if [ "$ENFORCE_EAGER" = "True" ]; then
-    export ENFORCE_EAGER="--enforce-eager"
-else
-    export ENFORCE_EAGER=""
-fi
-
-if [ "$ENABLE_PREFIX_CACHING" = "True" ]; then
-    export ENABLE_PREFIX_CACHING="--enable-prefix-caching"
-else
-    export ENABLE_PREFIX_CACHING=""
-fi
-
-if [ "$ENABLE_CHUNKED_PREFILL" = "True" ]; then
-    export ENABLE_CHUNKED_PREFILL="--enable-chunked-prefill"
-else
-    export ENABLE_CHUNKED_PREFILL=""
-fi
-
-if [ -z "$MAX_NUM_BATCHED_TOKENS" ]; then
-    export MAX_NUM_BATCHED_TOKENS=""
-else
-    export MAX_NUM_BATCHED_TOKENS="--max-num-batched-tokens=$MAX_NUM_BATCHED_TOKENS"
-fi
-
-# Activate vllm venv
-if [ "$VENV_BASE" = "singularity" ]; then
-    export SINGULARITY_IMAGE=/model-weights/vec-inf-shared/vector-inference_latest.sif
-    export VLLM_NCCL_SO_PATH=/vec-inf/nccl/libnccl.so.2.18.1
-    module load singularity-ce/3.8.2
-    singularity exec $SINGULARITY_IMAGE ray stop
-    singularity exec --nv --bind ${MODEL_WEIGHTS}:${MODEL_WEIGHTS} $SINGULARITY_IMAGE \
-    python3.10 -m vllm.entrypoints.openai.api_server \
-    --model ${MODEL_WEIGHTS} \
-    --served-model-name ${MODEL_NAME} \
-    --host "0.0.0.0" \
-    --port ${vllm_port_number} \
-    --tensor-parallel-size ${SLURM_GPUS_PER_NODE} \
-    --dtype ${DATA_TYPE} \
-    --max-logprobs ${MAX_LOGPROBS} \
-    --trust-remote-code \
-    --max-model-len ${MAX_MODEL_LEN} \
-    --max-num-seqs ${MAX_NUM_SEQS} \
-    --gpu-memory-utilization ${GPU_MEMORY_UTILIZATION} \
-    --compilation-config ${COMPILATION_CONFIG} \
-    --task ${TASK} \
-    ${MAX_NUM_BATCHED_TOKENS} \
-    ${ENABLE_PREFIX_CACHING} \
-    ${ENABLE_CHUNKED_PREFILL} \
-    ${ENFORCE_EAGER}
-
-else
-    source ${VENV_BASE}/bin/activate
-    python3 -m vllm.entrypoints.openai.api_server \
-    --model ${MODEL_WEIGHTS} \
-    --served-model-name ${MODEL_NAME} \
-    --host "0.0.0.0" \
-    --port ${vllm_port_number} \
-    --tensor-parallel-size ${SLURM_GPUS_PER_NODE} \
-    --dtype ${DATA_TYPE} \
-    --max-logprobs ${MAX_LOGPROBS} \
-    --trust-remote-code \
-    --max-model-len ${MAX_MODEL_LEN} \
-    --max-num-seqs ${MAX_NUM_SEQS} \
-    --gpu-memory-utilization ${GPU_MEMORY_UTILIZATION} \
-    --compilation-config ${COMPILATION_CONFIG} \
-    --task ${TASK} \
-    ${MAX_NUM_BATCHED_TOKENS} \
-    ${ENABLE_PREFIX_CACHING} \
-    ${ENABLE_CHUNKED_PREFILL} \
-    ${ENFORCE_EAGER}
-fi

vec_inf-0.5.0.dist-info/RECORD

@@ -1,17 +0,0 @@
-vec_inf/README.md,sha256=dxX0xKfwLioG0mJ2YFv5JJ5q1m5NlWBrVBOap1wuHfQ,624
-vec_inf/__init__.py,sha256=bHwSIz9lebYuxIemni-lP0h3gwJHVbJnwExQKGJWw_Q,23
-vec_inf/find_port.sh,sha256=bGQ6LYSFVSsfDIGatrSg5YvddbZfaPL0R-Bjo4KYD6I,1088
-vec_inf/multinode_vllm.slurm,sha256=V01ayfgObPdxbQqhYvCbNIx0zqpLurDxZhS0UHYNFi0,5210
-vec_inf/vllm.slurm,sha256=VMMTdVUOtX4-Yv43yzgKiEpE56fMwuR0KOLf3Dar_S0,2884
-vec_inf/cli/__init__.py,sha256=5XIvGQCOnaGl73XMkwetjC-Ul3xuXGrWDXdYJ3aUzvU,27
-vec_inf/cli/_cli.py,sha256=8Gk4NRbrY2-3EX0S8_-1UOmGfahzqX0A2sJNVcL7OL8,6525
-vec_inf/cli/_config.py,sha256=pb2ERbxZoRZBa9Ie7-jlzyQiiXYZgqUetbw13Blryho,2841
-vec_inf/cli/_helper.py,sha256=9niBuFoaJfeP2yRHKcrkia4rwCZbAfoz-4s2MVCwA9w,26871
-vec_inf/cli/_utils.py,sha256=i3mffIJ-wBVpe3pz0mYa2W5J42yNRmUG2tQwABhQJDQ,5365
-vec_inf/config/README.md,sha256=3MYYY3hGw7jiMR5A8CMdBkhGFobcdH3Kip5E2saq_T4,18609
-vec_inf/config/models.yaml,sha256=p__omZEmoF93BtVQXQia43xvPX88ALrBT1tMiY2Bdhk,28787
-vec_inf-0.5.0.dist-info/METADATA,sha256=khBhIsW5hFjrp-ZQFU3wa5htLVkp4CH3JnrhLXeHfss,10228
-vec_inf-0.5.0.dist-info/WHEEL,sha256=qtCwoSJWgHk21S1Kb4ihdzI2rlJ1ZKaIurTj_ngOhyQ,87
-vec_inf-0.5.0.dist-info/entry_points.txt,sha256=uNRXjCuJSR2nveEqD3IeMznI9oVI9YLZh5a24cZg6B0,49
-vec_inf-0.5.0.dist-info/licenses/LICENSE,sha256=mq8zeqpvVSF1EsxmydeXcokt8XnEIfSofYn66S2-cJI,1073
-vec_inf-0.5.0.dist-info/RECORD,,