[Python] How to include image as part of user prompt in haystack 2.X?

I have a great pipeline of chatbot using Haystack. I am referring to the haystack docs to create a pipeline, here is the example of the pipeline using prompt builder:

from haystack import Pipeline
from haystack.utils import Secret
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.components.builders.prompt_builder import PromptBuilder
from haystack.components.generators import OpenAIGenerator
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack import Document

docstore = InMemoryDocumentStore()
docstore.write_documents([Document(content="Rome is the capital of Italy"), Document(content="Paris is the capital of France")])

query = "What is the capital of France?"

template = """
Given the following information, answer the question.

Context:
{% for document in documents %}
{{ document.content }}
{% endfor %}

Question: {{ query }}?
"""
pipe = Pipeline()

pipe.add_component("retriever", InMemoryBM25Retriever(document_store=docstore))
pipe.add_component("prompt_builder", PromptBuilder(template=template))
pipe.add_component("llm", OpenAIGenerator(api_key=Secret.from_token("YOUR_OPENAI_API_KEY")))
pipe.connect("retriever", "prompt_builder.documents")
pipe.connect("prompt_builder", "llm")

res=pipe.run({
"prompt_builder": {
"query": query
},
"retriever": {
"query": query
}
})

print(res)


now I want to add new feature to my chatbot. I want the user to be able to add an image when they are asking some questions. Like the ChatGPT or Google Gemini, where you can add an image and ask the chatbot a question related to the image.

I have tried to add the image as in prompt builder template, and it didn't work. I changed the template to something like this:

template = """
Given the following information, answer the question.

Context:
{% for document in documents %}
{{ document.content }}
{% endfor %}

{% if image %}
{{ image }}
{% else %}
No image provided.
{% endif %}

Question: {{ query }}?
"""


Then pass the image like this:

res=pipe.run({
"prompt_builder": {
"query": query,
"image": image
},
"retriever": {
"query": query
}
})


I think it doesn't work because I need to explicitly tell the OpenAI that I included the image in the user prompt. So, How can I add the image as a user question in the haystack pipeline?

Continue reading...