[Python] LLM structured output sometimes incomplete or incoherent when generating JSON (Python...

I'm working on a Python application using LangChain where my agents generate structured outputs, in strict JSON format.

Here’s my problem: ➔ Sometimes the LLM (chat model) generates incoherent content either in the middle or at the end of the expected JSON structure.

Examples of incoherences:

• Incomplete JSON (e.g., missing closing braces)
• Extra irrelevant characters like random numbers, empty lines, or repeated sections
• Unexpected early stopping without completing the JSON structure

This happens even though:

• I specify a clear JSON response_schema
• I use streaming=True to display tokens progressively
• I instruct the model in the prompt to output only valid JSON, with no extra text
• I limit the number of output tokens with max_tokens

Example of expected output

{
"is_valid": true,
"overall_score": 8,
"validation_summary": "The content is clear and technically accurate.",
"strengths": ["Good explanations", "Accurate formulas"],
"areas_for_improvement": ["Simplify some parts"],
"technical_accuracy": {
"score": 9,
"issues": [],
"suggestions": []
}
}


Example of what I sometimes get:

{
"is_valid": true,
"overall_score": 8,
"validation_summary": "The content is clear and technically accurate.",
"strengths": ["Good explanations", "Accurate formulas"],
"areas_for_improvement": ["Simplify some


As you can see, the output gets broken and random tokens are generated inside the JSON structure.

What I have already tried:

• Lowering the temperature (e.g., 0.2) to reduce randomness
• Adding explicit stop instructions ("End output after closing }")
• Using post-parsing attempts to correct slightly invalid JSON
• Monitoring the stream to detect closing braces
• Setting max_tokens limits

What I'm looking for:

• A clean and reliable method to detect when the JSON is complete during streaming
• Strategies or patterns to force the model to terminate properly once the JSON structure is finished
• Any best practices or recommended libraries to secure structured outputs better (e.g., Guardrails, Outlines, etc.)

I'm open to ideas whether they involve:

• Better prompt engineering
• Post-processing techniques
• Changes in the way I call the model through LangChain

Thanks a lot for your help!

Continue reading...