[Python] Claude/Sonnet Python API - more tokens freezes, less tokens truncates

My prompt was to create a blog with 5 sections covering 5 different individuals in an industry niche.

When I had max_tokens= 300 (and then 1000), it ran quickly, but the result was not complete.

When I tried max_tokens=1500 (or 3000), then the program freezes, and doesn't come back after even about 5 minutes. I asked Claude what to do, and it provided this code to retry, but anytime I'm over 1500 tokens, I get the timeout.

def get_claude_response(prompt, max_retries=3, initial_max_tokens=2000, timeout=60):
for attempt in range(max_retries):
try:
message = client.messages.create(
model="claude-3-sonnet-20240229",
max_tokens=initial_max_tokens * (attempt + 1), # Increase max_tokens on each retry
messages=[
{"role": "user", "content": prompt}
],
timeout=timeout # Set a timeout for the API call
)
return message.content[0].text
except anthropic.APITimeoutError:
print(f"Timeout occurred. Retrying with increased max_tokens. Attempt {attempt + 1}/{max_retries}")
time.sleep(2) # Wait for 2 seconds before retrying
except Exception as e:
print(f"An error occurred: {str(e)}")
return None

print("Max retries reached. Unable to get a complete response.")
return None


# Example usage
prompt = "details here..."

print("Prompt:")
print(prompt)


response = get_claude_response(prompt)

if response:
print(response)
else:
print("Failed to get a response from Claude.")

I asked Claude itself for a solution and it came up with a method to call and get back chunks. Although it worked, it was very slow and was 5 calls, which might be costing me more in the long run than one call with more tokens.

def call_claude_sonnet_in_chunks(prompt, max_chunks=10, tokens_per_chunk=500, timeout=55):
full_response = ""
messages = [{"role": "user",
"content": prompt}]

for i in range(max_chunks):
try:
print ("Call Claude:Sonnet i=" + str(i) + " tokens_per_chunk=" + str(tokens_per_chunk))
message = client.messages.create(
model="claude-3-sonnet-20240229",
max_tokens=tokens_per_chunk,
messages=messages,
timeout=timeout
)

chunk = message.content[0].text
full_response += chunk

# Check if the response is complete
if chunk.strip().endswith('.'):
break

# Prepare for the next chunk
messages.append({"role": "assistant", "content": chunk})
messages.append({"role": "user", "content": "Please continue your previous response."})

time.sleep(1) # Small delay between requests
except anthropic.APITimeoutError:
print(f"Timeout occurred on chunk {i + 1}. Moving to next chunk.")
except Exception as e:
print(f"An error occurred: {str(e)}")
break

return full_response.strip()

Continue reading...