[Python] RuntimeError when trying to run suno/bark-small on GPU

When I run:

from transformers import AutoProcessor, BarkModel
import os
from scipy.io.wavfile import write as write_wav

CUDA_VISIBLE_DEVICES=0
os.environ["SUNO_OFFLOAD_CPU"] = "True"
os.environ["SUNO_USE_SMALL_MODELS"] = "True"

def bark():
processor = AutoProcessor.from_pretrained("suno/bark-small")
model = BarkModel.from_pretrained("suno/bark-small")

model.enable_cpu_offload()
model = model.to_bettertransformer()
model.to('cuda')

voice_preset = "v2/en_speaker_6"

inputs = processor('hello, i hope you are doing well', voice_preset=voice_preset)

audio_array = model.generate(**inputs)
audio_array = audio_array.cpu().numpy().squeeze()

sample_rate = model.generation_config.sample_rate
write_wav("/home/rocky/chatout/test.wav", rate=sample_rate, data=audio_array)

bark()


I get this error as a return:

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument test_elements in method wrapper_CUDA_isin_Tensor_Tensor)

I have attempted to offload the model to GPU multiple different ways yet I can't seem to get this error to stop occurring.

Continue reading...