[Python] Sending Chunks to Mistral API and Handling Streaming Responses

I'm working on a project where I need to send chunks of HTML content to the Mistral API and process the responses to generate slide content. I’ve written a script to handle this, but I'm running into issues with chunking and streaming responses.

The Problem


I need to send HTML content in chunks to the Mistral API and receive the responses for all chunks in one go. My code processes the content in chunks and sends them to the Mistral API using streaming. However, I'm encountering issues where the responses are not properly written to my output file.

Error Details


Here’s a sample of the output I get:

data: {"id":"217be063381c4fa0806d48018b99ade8","object":"chat.completion.chunk","created":1726290938,"model":"mistral-large-latest","choices":[{"index":0,"delta":{"content":" weather"},"finish_reason":null,"logprobs":null}]} data: {"id":"217be063381c4fa0806d48018b99ade8","object":"chat.completion.chunk","created":1726290938,"model":"mistral-large-latest","choices":[{"index":0,"delta":{"content":" patterns"},"finish_reason":null,"logprobs":null}]} data: {"id":"217be063381c4fa0806d48018b99ade8","object":"chat.completion.chunk","created":1726290938,"model":"mistral-large-latest","choices":[{"index":0,"delta":{"content":"."},"finish_reason":null,"logprobs":null}]} data: {"id":"217be063381c4fa0806d48018b99ade8","object":"chat.completion.chunk","created":1726290938,"model":"mistral-large-latest","choices":[{"index":0,"delta":{"content":""},"finish_reason":"stop","logprobs":null}],"usage":{"prompt_tokens":238,"total_tokens":890,"completion_tokens":652}} data: [DONE]

As you can see, the output is not correctly written to my output.html file, and the final response isn't as expected.

What I've Tried

• 
  I've implemented chunking using BeautifulSoup to handle the HTML content.
  
  
• 
  I’m using the Mistral API for streaming responses.
  

Code


Here’s the code I’m using:

from lxml import etree
import re
import sys
from bs4 import BeautifulSoup
from config import api_key
from mistralai import Mistral

class FileRW:
@staticmethod
def input_reader(input_file):
with open(input_file, 'r', encoding='utf-8') as html_file:
soup = BeautifulSoup(html_file, 'lxml')
for tag in soup.find_all(True):
tag.attrs = {}
n_f_s = "Not For Sale."
cengage_text = "© Cengage Learning Asia."
number_pattern = re.compile(r'\d{1,2}/\d{1,2}/\d{2,4}|\d{1,2}:\d{2}\s?[AP]M|\d+')
for tag in soup.find_all('p'):
tag_text = tag.get_text(strip=True)
if (n_f_s in tag_text or cengage_text in tag_text or number_pattern.search(tag_text)):
tag.decompose()
return str(soup)

@staticmethod
def chunk_generator(content, max_length, overlap_length=200):
soup = BeautifulSoup(content, 'html.parser')
chunks = []
current_chunk = ''
current_length = 0
elements = soup.body.children if soup.body else soup.children
for element in elements:
element_str = str(element)
element_length = len(element_str)
if current_length + element_length > max_length:
if current_chunk:
chunks.append(current_chunk[:max_length + overlap_length])
current_chunk = element_str
current_length = element_length
else:
current_chunk += element_str
current_length += element_length
if current_chunk:
chunks.append(current_chunk)
for chunk in chunks:
yield str(BeautifulSoup(chunk, 'html.parser'))

@staticmethod
def output_writer(content, output_file):
lines = content.splitlines()
if lines and '```html' in lines[0]:
lines.pop(0)
if lines and '```' in lines[-1]:
lines.pop()
cleaned_content = '\n'.join(lines).replace('●', '')
with open(output_file, 'w', encoding='utf-8') as output:
output.write(cleaned_content)

class MistralProcessor:
@staticmethod
def call_mistral_api(api_key, input_content, is_final_chunk=False):
client = Mistral(api_key=api_key)
model = "mistral-large-latest"
if not is_final_chunk:
messages = [
{"role": "user", "content": input_content + "\n\nI do have several chunks so please wait for me to provide other chunks"}
]
else:
prompts = MistralProcessor.prepared_prompts()
messages = [
{"role": "system", "content": prompts},
{"role": "user", "content": input_content}
]
try:
chat_response = client.chat.complete(
model=model,
messages=messages,
temperature=0.2,
max_tokens=10000,
top_p=1,
stream=True,
random_seed=1337
)
response_content = ""
for chunk in chat_response:
if chunk['choices'][0]['delta'].get('content'):
response_content += chunk['choices'][0]['delta']['content']
elif chunk['choices'][0]['finish_reason'] == 'stop':
break
return response_content
except Exception as e:
print(f"Error: {e}")
return None

@staticmethod
def prepared_prompts():
return (
'''
I will provide you chunks of HTML content from a study textbook. Your task is to transform each chunk into slides that summarize the information into bullet points, while keeping it clear and concise.

**Instructions**:

1. Organize the slides according to the structure provided in the textbook.
2. Use bullet points for key concepts and brief explanations where needed.
3. If the chunk is part of a section, build on previous content without repeating it.
4. Where examples are provided, include them in a simple format.
5. Ensure that the output remains consistent throughout the chunks.

**Final Requirements**:
- Keep the overall presentation between 45-55 slides.
- Only include necessary images, and skip those that are irrelevant.
'''
)

@staticmethod
def process(input_file, output_file):
max_length = 1000 # Adjust based on token count
overlap_length = 200 # Amount of content overlap to maintain continuity
html_content = FileRW.input_reader(input_file)
input_generator = FileRW.chunk_generator(html_content, max_length, overlap_length)
all_responses = []
previous_chunk = ''
chunk_count = 0
total_chunks = sum(1 for _ in FileRW.chunk_generator(html_content, max_length, overlap_length))
input_generator = FileRW.chunk_generator(html_content, max_length, overlap_length)
for input_content in input_generator:
is_final_chunk = (chunk_count == total_chunks - 1)
combined_content = previous_chunk + input_content
chat_response = MistralProcessor.call_mistral_api(api_key, combined_content, is_final_chunk)
if chat_response:
all_responses.append(chat_response)
previous_chunk = input_content[-overlap_length:]
chunk_count += 1
full_response = ''.join(all_responses)
FileRW.output_writer(full_response, output_file)

def main(input_file, output_file):
MistralProcessor.process(input_file, output_file)

if __name__ == "__main__":
if len(sys.argv) < 3:
print("Usage: python script.py <input_file> <output_file>")
sys.exit(1)

input_file = sys.argv[1]
output_file = sys.argv[2]
main(input_file, output_file)


What I Need Help With How can I properly handle the streaming responses so that all chunks are combined correctly?

Is there a better approach to sending chunks and receiving the final response in one go?

Any advice or suggestions on how to resolve this issue would be greatly appreciated. Thanks!

Continue reading...