[Python] How to make sure my AI code correctly uses Google search?

I am using the following (straightforward) code to run a prompt with some text and images to Google Gemini. The system instruction is quite extensive but clearly states to 1) use the tool, and 2) provide the thinking results and if it actually used the tool in its work.

parts = []
for image in images:
with open(image, "rb") as f:
parts.append(types.Part.from_bytes(mime_type="image/jpeg", data=f.read()))

parts.append(types.Part.from_text(text=text_part))

contents = [
types.Content(
role="user",
parts=parts
),
]
tools = [
types.Tool(googleSearch=types.GoogleSearch()),
]
generate_content_config = types.GenerateContentConfig(
thinking_config = types.ThinkingConfig(
thinking_budget=-1,
),
tools=tools,
system_instruction=[
types.Part.from_text(text=system_instruction),
],
)

raw_result = ""
for chunk in self.client.models.generate_content_stream(
model=self.model,
contents=contents,
config=generate_content_config,
):
if chunk.text:
raw_result += chunk.text


The system instruction states (among other things):

You can use the provided tool to enhance the information. In your reasoning, specify if you use the tool, which searches you did and what their results were.


Return your reasoning and the final JSON result in the format below. All fields must be populated with a value, even if that value is an empty string "".
​

However, the results are always only the JSON (wrapped in ````) and the provided URLs are all nonsense 404s or connection refused.

I tried checking the documentation and also Gemini itself, but it just keeps spitting out nonsense when it comes to its Google search.

Continue reading...