[Python] What should I set as LoRA target_modules for each stage of continued pretraining on...

I would like to perform continued pretraining of Qwen2.5-VL-Instruct using Unsloth + LoRA, following a three-stage training process:

• Stage 1: Train only the projector (Alignment)
• Stage 2: Train both the projector and the LLM (Pretraining)
• Stage 3: Train everything — the vision encoder, projector, and LLM (Supervised Fine-tuning)

In each stage, I want to use LoRA. What should I set as target_modules for each phase to correctly apply LoRA to the appropriate components?

Specifically:

How do I restrict LoRA to only the projector in Stage 1?

How do I apply LoRA to both the projector and LLM in Stage 2?

How do I include the vision encoder as well in Stage 3?

If you have experience applying LoRA to vision-language models with Unsloth or other tools, I’d appreciate any insights or code snippets.

Continue reading...