[Python] Why does GPT4All not strictly follow the JSON format specified in the prompt and add...

I’m using GPT4All in a Python script to classify emails by category, priority, and response requirement. Despite a clear prompt specifying a strict JSON format and instructions to exclude additional text, the LLM frequently outputs invalid JSON, includes comments or explanations, and sometimes adds unwanted placeholders like
or " ". This behavior makes it difficult to process the LLM’s output programmatically.

I used the following MODEL_SETTINGS to ensure the LLM generates deterministic and focused outputs:

MODEL_SETTINGS = {
"temp": 0.0, # Ensures deterministic outputs
"top_p": 0.5, # Focuses on the highest probability tokens
"top_k": 1, # Chooses the top token every time
"repeat_penalty": 2.5, # Penalizes repetitive sequences
}

prompt = (
f"<user>\n"
f"You are an AI system that only outputs JSON objects. Your task is to classify the following email into a predefined JSON format.\n\n"
f"Email details:\n"
f"Subject: \"{subject}\"\n"
f"Sender: \"{sender}\"\n\n"
f"### Output Format\n"
f"Respond only in this JSON format:\n"
f"{{\n"
f" \"Category\": \"<Work, School, or Shopping>\",\n"
f" \"Priority\": \"<Important or Normal>\",\n"
f" \"RequiresResponse\": \"<Yes or No>\"\n"
f"}}\n\n"
f"### Example Response\n"
f"{{\n"
f" \"Category\": \"Work\",\n"
f" \"Priority\": \"Normal\",\n"
f" \"RequiresResponse\": \"No\"\n"
f"}}\n\n"
f"### Important Instructions\n"
f"1. Output *only* a single JSON object in the exact format provided above.\n"
f"2. *Do not include any additional text, comments, explanations, or blank lines.*\n"
f"3. Your response *must* be a valid JSON object with proper capitalization, syntax (double quotes, etc.), and no trailing commas.\n"
f"4. If you cannot classify the email, leave the values blank (e.g., \"Category\": \"\") but maintain the JSON structure.\n\n"
f"Now classify the email:\n"
f"%1<end>\n"
f"<assistant>\n"
f"%2<end>\n"
)


I expected the LLM to output a valid JSON object strictly adhering to the provided format, such as:

{
"Category": "Work",
"Priority": "Normal",
"RequiresResponse": "No"
}


What actually happened?

The LLM often outputs responses with issues like: • Extra text, explanations, or comments. • Invalid JSON structure with missing quotes or keys. • Unnecessary placeholders such as
, " ", or %1.

Continue reading...