[Python] "You have a version of `bitsandbytes` that is not compatible with 4bit inference and...

I am now trying to finetune a llama3 model. I am using unsloth,

from unsloth import FastLanguageModel


Then I load Llama3 model.

model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "unsloth/llama-3-8b-bnb-4bit",
max_seq_length = max_seq_length,
dtype = None,
load_in_4bit = True,
)


I am running my script on CS Code, and my python and script are on WSL. My system info is as below:

==((====))== Unsloth: Fast Llama patching release 2024.5
\\ /| GPU: NVIDIA GeForce RTX 4090. Max memory: 23.988 GB. Platform = Linux.
O^O/ \_/ \ Pytorch: 2.1.0+cu121. CUDA = 8.9. CUDA Toolkit = 12.1.
\ / Bfloat16 = TRUE. Xformers = 0.0.22.post7. FA = False.
"-____-" Free Apache license: http://github.com/unslothai/unsloth


Now I run into this error:

---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[4], line 2
1 # 2. Load Llama3 model
----> 2 model, tokenizer = FastLanguageModel.from_pretrained(
3 model_name = "unsloth/llama-3-8b-bnb-4bit",
4 max_seq_length = max_seq_length,
5 dtype = None,
6 load_in_4bit = True,
7 )

File ~/miniconda/envs/llama3/lib/python3.9/site-packages/unsloth/models/loader.py:142, in FastLanguageModel.from_pretrained(model_name, max_seq_length, dtype, load_in_4bit, token, device_map, rope_scaling, fix_tokenizer, trust_remote_code, use_gradient_checkpointing, resize_model_vocab, *args, **kwargs)
139 tokenizer_name = None
140 pass
--> 142 model, tokenizer = dispatch_model.from_pretrained(
143 model_name = model_name,
144 max_seq_length = max_seq_length,
145 dtype = dtype,
146 load_in_4bit = load_in_4bit,
147 token = token,
148 device_map = device_map,
149 rope_scaling = rope_scaling,
150 fix_tokenizer = fix_tokenizer,
151 model_patcher = dispatch_model,
152 tokenizer_name = tokenizer_name,
...
96 "You have a version of `bitsandbytes` that is not compatible with 4bit inference and training"
97 " make sure you have the latest version of `bitsandbytes` installed"
98 )

ValueError: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules in 32-bit, you need to set `load_in_8bit_fp32_cpu_offload=True` and pass a custom `device_map` to `from_pretrained`. Check https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu for more details.


`

Would anyone please help?

Continue reading...