I am creating a chatbot that uses LLM - gpt-4o. I provide approximately 200 filtered tickets, that has data of service now tickets. I ask question like Show me my incidents. It takes 2 mins to generate response for 200 tickets in context. I don't do streaming of response right now I know streaming might help as per user experience but still for now what to do for improving time if response generation. Context contains data like this; Number: Short Description: Issue details : Status : Open data : Closed date : Code : response = chain.invoke({ "question": query, }) Continue reading...
