Wondering if anyone has been able to get streaming working with Llamaapi in Python. It's probably that I am not calling asyncio correctly as I am pretty new to that. Note tthat there isn't space to give the full code, so these are snippets. I've got: async def reply_async(args): async for chunk in client.run(args): print(f"DEBUG: Async ch: {chunk}") # assign reply BTW, that run call returns a non-iterable async generator object, so this has to be async for, in case you are wondering why I am not just iterating normally like one would in Openai. Here's how I call it: reply = asyncio.run(reply_async(args)) This gives back a single "chunk" which is actually several lines containing the ENTIRE response instead of line by line, so it's basically useless as an async call. You an see that the "DEBUG" label only occurs once at the top: 19:25:04 DEBUG:Async ch: data: {"id":"91b68106f9b322d0","created":1741145104,"model":"llama3.1-8b","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Let","token_id":10267},"text":"Let"}]} data: {"id":"91b68106f9b322d0","created":1741145104,"model":"llama3.1-8b","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"\u0027s","token_id":596},"text":"\u0027s"}]} data: {"id":"91b68106f9b322d0","created":1741145104,"model":"llama3.1-8b","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" try","token_id":1456},"text":" try"}]} and so on. Like I said, I am probably calling asyncio wrong but I can't find any examples of this working. Continue reading...
