[Python] AttributeError: 'CLIPImageProcessor' object has no attribute 'patch_size' when...

I have fine-tuned a LLaVA (Large Language and Vision Assistant) model on Google Colab and saved it to my Google Drive. Here’s how I saved the model:

'''

from google.colab import drive
drive.mount('/content/drive', force_remount=True)
import os

save_path = "/content/drive/MyDrive/fineTune model1/LLaVA-med-MAKAUT_v1"
os.makedirs(save_path, exist_ok=True)

trainer.model.save_pretrained(save_path)
trainer.tokenizer.save_pretrained(save_path)
processor.image_processor.save_pretrained(save_path)


'''

After saving, my Google Drive folder contains the following files:

• 
  README.md
  
  
• 
  adapter_model.safetensors
  
  
• 
  adapter_config.json
  
  
• 
  tokenizer_config.json
  
  
• 
  special_tokens_map.json
  
  
• 
  added_tokens.json
  
  
• 
  tokenizer.model
  
  
• 
  tokenizer.json
  
  
• 
  preprocessor_config.json
  
  
• 
  config.json
  

However, when I try to load the model for testing, I get an AttributeError related to patch_size:

'''

import torch
from PIL import Image
from transformers import LlavaProcessor, LlavaForConditionalGeneration, CLIPImageProcessor

model_path = "/content/drive/MyDrive/fineTune model/LLaVA-med-MAKAUT_v1"
processor1 = LlavaProcessor.from_pretrained(model_path)


'''

Checking patch size from the model's vision_config


'''

patch_size = new_model_v1.config.vision_config.patch_size
print("Patch size:", patch_size)


'''

Output: Patch size: 14

Error Occurs Here :


'''

print(processor1.image_processor.patch_size)


'''

Error Message:

'''

AttributeError: 'CLIPImageProcessor' object has no attribute 'patch_size'


'''

What I Have Tried: Ensuring that the model is properly saved and loaded. Confirming that the patch size is present in the model's vision configuration (patch_size: 14). Attempting to manually set patch_size:

'''

processor1.image_processor.patch_size = 14


'''

However, this doesn't seem to be the right approach since CLIPImageProcessor doesn’t have this attribute.

Questions:

1. Why does CLIPImageProcessor lack the patch_size attribute even though it is defined in the model’s vision_config?
2. What is the correct way to ensure that the LLaVA processor aligns with the fine-tuned model’s configuration, especially concerning patch_size?
3. Is there a recommended way to properly load and utilize the fine-tuned LLaVA model along with its processor for inference in Colab?

Continue reading...