Voice AI agents
AI phone agents that answer calls, understand the caller and reply from your own data. Built with Twilio Programmable Voice, a FastAPI server and OpenAI, the Claude API or Llama 3, with AWS Transcribe for voice messages.
What is it?
A voice AI agent is an AI agent that talks instead of types. A phone number sends each call to your server, the caller's speech is turned into text, a large language model (LLM) writes the reply, and the reply is spoken back.
Why use it?
Callers get an answer at any hour without waiting for a person, and the agent answers from your own products, policies and records. Calls it should not handle can be passed to a person.
How I build it
- Point a Twilio number at a FastAPI webhook that answers with TwiML.
- Turn the caller's speech into text and send it, with the recent call history, to the LLM.
- Answer from your own data with retrieval (RAG) through LlamaIndex.
- Convert voice messages to text with AWS Transcribe.
- Validate Twilio's request signatures and hand calls to a person with <Dial> when needed.
Why hire me?
I built Xobot (xobot.co) end to end: a no-code platform where businesses run AI voice agents that take phone calls through Twilio Programmable Voice. I also wrote a step-by-step guide to building an AI phone agent with Twilio and FastAPI.