Back to work
Confidentiality note: The public diagram shows the technical call flow only. Phone numbers, credentials, recordings, and customer data are not published.
Real-time voice + telephony
AWS Nova Sonic Voice Agent
A bidirectional speech-to-speech agent for inbound and outbound Twilio calls, with live tool use and Bedrock-grounded context.
Bidirectional audioInbound + outbound callsMid-call tool use
Problem
Phone-based service workflows need natural turn-taking, grounded answers, and reliable actions without forcing callers through rigid menu trees or disconnected speech services.
Approach
- Connected Twilio Media Streams to Nova Sonic so caller audio and model speech could move bidirectionally during one live session.
- Added Bedrock retrieval for approved knowledge and tool schemas for workflow actions during the call.
- Kept telephony events, model events, and tool results separated so sessions could be inspected and extended without coupling every concern.
Architecture
- Twilio initiates or receives the phone call and opens a media WebSocket.
- The Fastify service converts telephony audio into the stream format expected by Nova Sonic.
- Nova Sonic manages speech understanding, response generation, and spoken output in one bidirectional session.
- Knowledge retrieval and tool handlers provide grounded context and return structured results mid-call.
Production
- Audio codec and sample-rate handling preserve intelligibility across the telephony boundary.
- Session and event separation make disconnects, tool failures, and call state easier to diagnose.
- Human escalation remains an explicit path for workflows that require an operator.
Result
The prototype demonstrated a complete voice-agent loop from dial and live conversation through retrieval and tool execution, rather than a browser-only speech demo.
Stack