[Python] Ollama model keep in memory and prevent unloading between requests (keep_alive?)

No matter what I do from the terminal or in code, the agent requests to Ollama models take 15–25 seconds each time on my local M2 MacBook Pro.

I am pretty sure this is not a hardware issue because the model is lightning fast when used from the terminal.

I am using Python's module to load gemma3.4b via Ollama. Tried different models – similar result.

Tried calling via http – not much improvement.

So, my conclusion is that the model keeps unloading from memory each time and loads on new requests.

I've looked for keep_alive = option and found that some say it defaults to -1. And it can be changed when running from CLI, but I could not find docs on how to use it in Python, even though there is, apparently, a fixed bug on adding this feature.

https://github.com/ollama/ollama-python/pull/31

Here is my code:


import json
import asyncio

from library import Model
from pydantic import BaseModel
import ollama


class ModelOllama(Model):
def __init__(self, name: str):
super().__init__(name)

def _get_client(self):
return ollama

def _format(self, schema: BaseModel):
return schema.model_json_schema()

async def __call__(self,
prompt : str,
response_schema : BaseModel,
role : str = 'user',
temperature : float = 0.0
):
params = {
'model' : self.name,
'messages' : [{
'role' : role,
'content' : prompt
}],
'format' : self._format(response_schema),
'options' : {
'temperature': temperature,
# 'num_gpu': 1, # Use GPU acceleration if available
# 'num_thread': 6, # Use multiple threads
'keep_alive': 60 # Keep model loaded for 1 minute (60 seconds)
}
}

try:
response = await asyncio.to_thread(self.client.chat, **params)
output = json.loads(response['message']['content'])
except json.JSONDecodeError as e:
raise Exception(f'OllamaModel json parsing error: {e}')
except Exception as e:
raise Exception(f'OllamaModel LLM communication error: {e}')

return output

Continue reading...