[Python] When should I stop stuffing \~50k tokens into LLM prompts and move to MCP/tools...

I’ve built a Python agent that generates and executes code using a large, example-heavy prompt. Right now, I’m sending around 50k tokens on each run. It works, but I’m worried about scalability, latency, costs, and hitting rate limits. But I get good results sending 50k of tokens on each call... and am not concerned with API costs, yet.

Currently, I reload and resend the same static “reference examples” every attempt (generate → run → capture error → fix).

I originally dove right into MCP and my own server but resorted back to just sending 50k of context tokens on each API call because the results appeared better. The LLM agent found the solution much sooner and I think better results.

My questions:

• What are clear signals it’s time to move from huge prompts to MCP/tools?
• What best practices help most when making this shift (prompt splitting, caching, caps)?
• Any common pitfalls with MCP design or runaway tool usage?
• Favorite tricks for “token hygiene” (deduping, comment stripping, trimming examples)?
• If you were me, what’s the smallest next step to get most of the benefits?
• Testing on models with small context windows (IE., grok mini of 4k context) this obliviously doesn't work but some model I with large context windows it does not seem to be a big deal. So would costs only be my considerations?

Environment: Python agent calling Gemini, prompts = code examples + instructions, agent retries until success.

Continue reading...