[Python] Real-time call transcription using Twilio Media Streams and Deepgram in...

I've built a call-center application using Next.js for the frontend and FastAPI for the backend, integrated with Twilio for voice calls. The core features are working: users can initiate outbound calls from the frontend portal to any phone number, and the recipient receives the call, allowing real-time conversation via our dialer app (using Twilio's Voice SDK for browser softphone support). Now, I'm trying to extend this to include real-time transcription of the call audio (both caller and callee) using Twilio's Media Streams and Deepgram as the transcription service. I've set up a WebSocket endpoint in FastAPI to handle the media stream from Twilio and forward audio to Deepgram, but the transcription isn't working—the WebSocket doesn't seem to connect properly from Twilio, and no real-time transcripts appear on the frontend.

Configured a TwiML app in Twilio with a webhook URL pointing to my ngrok-tunneled FastAPI endpoint (e.g., https://1dad7cb6fcf4.ngrok-free.app/api/v1/twilio/voice). In the backend (twilio_simple.py), I'm generating TwiML that includes a verb to send both inbound and outbound audio tracks to a WebSocket URL derived from the webhook domain (e.g., wss://1dad7cb6fcf4.ngrok-free.app/api/v1/media-stream). Implemented a WebSocket handler in media_stream.py to receive Twilio's media stream, extract the CallSID from the 'start' event, decode base64 audio payloads, and forward them to Deepgram for transcription (with speaker diarization based on 'inbound'/'outbound' tracks). On the frontend (ConversationDisplay component in Next.js), I'm using a WebSocket connection to the backend (ws://localhost:8001/api/v1/dashboard/ws) to receive real-time transcript messages, with fallback polling to load conversations if the WebSocket fails. Tested with ngrok for public exposure, and confirmed basic calls work without transcription. Debugged the WebSocket URL (initially had an extra /media-stream in the path, which I corrected based on some articles).

Despite this, the media stream WebSocket doesn't connect—no logs for "Media stream WebSocket connected" in the backend, and Twilio's debugger shows no obvious errors for the voice webhook, but possibly failures for the WSS connection. Frontend shows "Disconnected" for the dashboard WebSocket, and no transcripts appear (even with polling fallback). Deepgram API key is set, but no audio is reaching it.

Backend: TwiML Generation in twilio_simple.py (for browser softphone calls):


`# ... (inside handle_voice_webhook)
if settings.DEEPGRAM_API_KEY and settings.TWILIO_WEBHOOK_URL:
try:
parsed_url = urlparse(settings.TWILIO_WEBHOOK_URL)
webhook_domain = parsed_url.netloc
stream_url = f"wss://{webhook_domain}/api/v1/media-stream" # Corrected path
stream = Stream(url=stream_url, track="both_tracks")
response.append(stream)
logger.info("✅ Media streaming enabled for call transcription")
except Exception as e:
logger.warning(f"❌ Failed to set up media streaming: {e}")

dial = response.dial(
action=f"{settings.TWILIO_WEBHOOK_URL}/api/v1/twilio/dial-status",
method='POST',
timeout=30,
caller_id=To,
record=True,
recording_status_callback=f"{settings.TWILIO_WEBHOOK_URL}/api/v1/twilio/recording-status",
recording_status_callback_event="completed"
)
dial.number(target_num, status_callback=..., status_callback_event=...)`



Backend: WebSocket Handler in media_stream.py:

`@router.websocket("/media-stream")
async def media_stream_websocket(websocket: WebSocket):
# Handles connection, extracts CallSID from 'start' event, forwards audio to Deepgram
# ... (full handler code as in context)`


Frontend: WebSocket in Next.js (ConversationDisplay):

`// Initializes WebSocket to ws://localhost:8001/api/v1/dashboard/ws
// Handles onmessage for 'conversation_message' type, updates messages state
// Fallback polling to /api/v1/dashboard/active-calls and conversations`


Logs/Errors:

Backend: "ℹ️ Media streaming disabled" if keys missing, or "❌ Failed to set up media streaming" with URL parse errors. No WebSocket connection logs from Twilio side. Frontend: "WebSocket disconnected: 1006" or polling fallback activates, but no real-time updates.

Environment:

FastAPI on port 8001, ngrok for tunneling. Twilio SDK, Deepgram SDK installed. Python 3.x, Next.js 14+.

Can anyone experienced with Twilio Media Streams, Deepgram, and WebSockets in FastAPI/Next.js help? Specifically, how to properly extract and transcribe real-time voice from both caller and callee? Am I missing something in the TwiML (e.g., placement) or WebSocket config? Alternatives like Twilio's built-in transcription work for post-call, but I need real-time. I'm fully blocked—any guidance or code fixes appreciated!

Continue reading...