[Python] How can run vLLM model on a multi GPU server

I want to run an LLM model on a multi-gpu server (4 GPU) using vLLM.

Here is the command line I run:

python3 -m vllm.entrypoints.api_server --model bjaidi/Phi-3-medium-128k-instruct-awq --quantization awq --dtype auto --gpu-memory-utilization 0.8


I got this error:

[rank0]: torch.OutOfMemoryError: Error in model execution
(input dumped to /tmp/err_execute_model_input_20240916-094901.pkl):
CUDA out of memory. Tried to allocate 4.38 GiB. GPU 0 has a total capacity
of 21.96 GiB of which 3.09 GiB is free.
Including non-PyTorch memory, this process has 18.87 GiB memory in use.
Of the allocated memory 18.58 GiB is allocated by PyTorch,
and 63.57 MiB is reserved by PyTorch but unallocated.
If reserved but unallocated memory is large try setting
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.
See documentation for Memory Management
(https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)


any hints ?

Continue reading...