[Python] LoRA Adapter Loading Issue with Llama 3.1 8B - Missing Keys Warning

I'm having trouble loading my LoRA adapters for inference after fine-tuning Llama 3.1 8B. When I try to load the adapter files in a new session, I get a warning about missing adapter keys:

/usr/local/lib/python3.11/dist-packages/peft/peft_model.py:599: UserWarning: Found missing adapter keys while loading the checkpoint: ['base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight', 'base_model.model.model.layers.0.self_attn.q_proj.lora_B.default.weight', 'base_model.model.model.layers.0.self_attn.k_proj.lora_A.default.weight',...

The warning continues with many similar missing keys. The model behaves as if it wasn't fine-tuned at all.

My setup:

• transformers 4.43.1
• peft 0.14.0
• bitsandbytes 0.45.4
• torch 2.6.0+cu124
• accelerate 0.31.0

After training, I saved the model and got these files:

1. adapter_model.safetensors
2. adapter_config.json
3. training_args.bin

Here's how I loaded the model and adapters:

# Load base model
base_model_id = "meta-llama/Llama-3.1-8B"
lora_weights = "path to weights"

bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16
)

base_model = AutoModelForCausalLM.from_pretrained(
pretrained_model_name_or_path=base_model_id,
quantization_config=bnb_config,
trust_remote_code=True,
token=True
)

# Set up tokenizer
eval_tokenizer = AutoTokenizer.from_pretrained(
base_model_id,
add_bos_token=True,
trust_remote_code=True,
use_fast=True
)
eval_tokenizer.pad_token = eval_tokenizer.eos_token

# Load LoRA adapters
ft_model = PeftModel.from_pretrained(base_model, lora_weights, output_loading_info=True)


Any ideas on what might be causing this issue or how to fix it?

Thanks in advance!

Continue reading...