I have created a fastAPI web interface to a local RAG powered question answer system. I have a local vector database populated with proprietary information. I first take the user question from the webpage and use it to retrieve results from chroma DB. Then I feed those results in as the context of a prompt to a local instance of mistral 7B. I would like to enforce a specific format of response from the LLM. What I have read about outlines sounds great, but I have not been able to get it to work in the context of streaming the output from the LLM. Has anyone had success with this? I’m currently using pydantic to specify an output format to fastAPI and validate the LLM response through an expected JSON object structure. This is an improvement, but not all the way to the strict enforcement that outlines provides using regular expressions. Continue reading...
