What I want to do is to adapt one of the models (Qwen/Qwen2.5-VL-3B-Instruct or Qwen/Qwen2.5-VL-7B-Instruct) to food domain. I may be wring in choosing the model, correct me in that case, but those both seem like they can be trained on an NVIDIA L4 GPU which is available in google colab. I am familiar with Python and neural networks but LLMs and especially VLMs are not something I studied at university. I have some recipes with step-by-step photos with text. They are currently structured like this: image of the final dish, dish description, first step, photo of the first step, second step, photo of the second step and so on. Each recipe is represented as a txt file with local paths to images instead of actual photos, they are stored separately. I have searched for a long time, even asked various LLMs, but still have no idea how the dataset for domain adaptation has to be structured and how a working code looks. I heard mentioning some libraries for QLoRA for fine-tuning language models, this is probably what I need. I'm afraid that this area is still new for LLMs to be good at. Please help me with a direct link to a page that explains how to do QLoRA on Visual Language Models or an explanation so that I understand. Also please provide feedback on the correctness of using an instruct model to adapt domain. Maybe I need to use a base model? The specifics of what I want this model to do after that are not important, for now I just want to perform domain adaptation for food and recipes. One of key points to keep in mind is that dataset looks like images and text mixed together, not just a couple of images in the beginning and then only text. The dataset contains 10000 examples. Continue reading...
