[Python] The finetuned IP-Adapter model fails to achieve effective results

The fine tuning environment is based on the open source and its instructions: https://github.com/tencent-ailab/IP-Adapter

I trained an IP-Adapter model for finetuning with image + prompt pairs as training data. However, the finetuned model does not reflect the expected result based on the prompt.

2 failed cases for train images and .json prompt file:

1. 
   Same prompt for daylight images (17 images): Eg, "image_file": "day_1.jpg", "text": "A city during the daytime with sunshine."
   
   Same prompt for night images (17 images): Eg, "image_file": "night_1.jpg", "text": "A city during night with light. No sunshine." Inference result: After training, input day images with prompt such as "A city during the daytime with sunshine." for inference, the generated output images are still day images, not night images.
   
   
2. 
   Train for thousands of images (1500 neutral face and 2500 sad face images): 4 different prompts for neutral images to represent neutral face expression. Eg,"image_file": "neu_1.jpg", "text": "The person's face has a neutral expression." 4 different prompts for sad images to represent sad face expression. Eg,"image_file": "sad_1.jpg", "text": "The person is showing a sorrowful expression."
   

Inference result: After training, apply neutral images with a prompt such as "A man is very sad.", or "The person is showing a sorrowful expression." , the generated output face images are still neutral and not sad at all.

Finetuning steps:

1. 
   Modify the code in tutorial_train_plus.py: replace accelerator.save_state(save_path)
   
   with accelerator.save_state(save_path, safe_serialization=False)
   
   
2. 
   Run the scripts below for fine tuning:
   
   accelerate launch --num_processes 2 --multi_gpu --mixed_precision "fp16"
   tutorial_train_plus.py
   --pretrained_model_name_or_path="stable-diffusion-v1-5/"
   --pretrained_ip_adapter_path="models/ip-adapter-plus_sd15.bin"
   --image_encoder_path="models/image_encoder/"
   --data_json_file="assets/prompt_image.json"
   --data_root_path="assets/images/train/"
   --mixed_precision="fp16"
   --resolution=512
   --train_batch_size=8
   --dataloader_num_workers=4
   --learning_rate=1e-04
   --weight_decay=0.01
   --output_dir="out_model"
   --num_train_epochs=300
   --save_steps=200
   
   
3. 
   Refer to the instructions in readme and convert pytorch.bin to ip-adapter.bin. In the inference file ip_adapter-plus_demo.py, Modify the original model: ip_ckpt = "models/ip-adapter-plus_sd15.bin"
   

to trained model ip_ckpt = "models/ip-adapter.bin"

1. run python3 ip_adapter-plus_demo.py for inference

Is there anything wrong on the process above, or the problem is on input images or prompts? In my case, how many dataset pairs (one image and one prompt) are generally required, at a minimum, to achieve effective results?

Continue reading...