[Python] How to extract bounding box from an image that contains the answer to a given...

I’m working on a task where I’m given:

• A question
• An image (which may contain text, tables, or visual content).

The goal is to identify the bounding box (coordinates) in the image that contains the answer to the question, drawing a box around the relevant region.

What I've tried:

• I used LLMs (OpenAI and Anthropic) with structured output to get the coordinates. However, the results were not reliable, accuracy was around 30%, and bounding boxes were often off or unrelated.
• I haven’t been able to find a consistent, accurate approach using just LLMs.

class BoundingBox(BaseModel):
top_left: List[int]
bottom_right: List[int]
content: str
confidence: float

class ImageAnalysisResult(BaseModel):
question: str
image_resolution: List[int]
bounding_boxes: List[BoundingBox]
overall_answer: str


response = client.responses.parse(
model=model,
temperature=0.0,
input=[
{
"role": "system",
"content": system_prompt,
},
{
"role": "user",
"content": [
{
"type": "input_text",
"text": user_prompt,
},
{
"type": "input_image",
"image_url": f"data:image/png;base64,{base64_image}",
"detail": "high",
},
],
},
],
text_format=ImageAnalysisResult,
)


What I’m looking for:

• A pretrained model or a combined NLP + CV2 based method that can locate the answer within the image using bounding box coordinates.
• Open to ideas using OCR + semantic similarity + vision models or any other efficient pipeline.

Any suggestions for models, libraries, or workflows that can help?

Thanks in advance!

Continue reading...