How can I ensure that the microphone ignores the bot's audio and only listens to the user's...

I'm working on a real-time voice chatbot in Angular 16 with a backend that uses Azure Speech Services for Text-to-Speech (TTS) and Speech-to-Text (STT). The goal is to have a seamless conversation where the bot:

Listens to the user via the microphone. Responds with synthesized audio. Stops playing audio and starts listening again if the user speaks while the bot's response is still playing. Here's my current setup:

Frontend: Angular 16 service and component handle microphone input and audio playback. Backend: Azure Speech Services convert text to speech. Speech Recognition: I'm using SpeechRecognition and SpeechSynthesis in the frontend. The challenge I'm facing is that when the bot plays its synthesized audio, the microphone recognizes the bot's own voice instead of the user's voice. This leads to inaccurate transcription and a disrupted user experience.

Issues: Echo problem: When the bot plays audio, the microphone picks up the bot's voice instead of waiting for the user. Interrupt mechanism: I want to stop audio playback if the user starts speaking while the bot's response is playing.

Using audio.onplaying and audio.onended events to control when the microphone is on or off. Attempting to filter out the bot's audio from the microphone input using Web Audio API. Questions: How can I ensure that the microphone ignores the bot's audio and only listens to the user's voice? Is there a way to detect user speech in real-time during bot audio playback and stop the audio dynamically? Are there any best practices or libraries for implementing a whispering feature for real-time voice bots using Angular and Azure Speech Services?

Continue reading...