How to build an AI phone agent with Twilio and FastAPI

How does an AI phone agent work with Twilio and FastAPI?

An AI phone agent answers a phone call, listens to the caller, sends what they said to a large language model (LLM), and speaks the reply back. With Twilio Programmable Voice, each step of the call is an HTTP webhook to your server. Your FastAPI app answers each webhook with TwiML, and the LLM decides what to say.

TwiML (Twilio Markup Language) is a small XML format that tells Twilio what to do on the call. Two TwiML verbs do most of the work here. <Gather input="speech"> listens and transcribes the caller, and <Say> turns text into speech.

The loop is simple. Twilio calls /voice/incoming when the call starts. Your app greets the caller inside a <Gather>. When the caller finishes speaking, Twilio posts the transcript to /voice/turn as SpeechResult, your app asks the LLM for a reply, and returns another <Gather> with that reply inside. The call continues turn by turn until someone hangs up.

In the no-code AI bot framework I built, Twilio Programmable Voice handled phone calls in the same way, with a FastAPI backend and a configurable LLM behind it.

What do you need before you start?

You need a Twilio account with a voice-capable phone number, Python 3.10 or newer, and an LLM you can call from Python. Install the packages below. FastAPI needs python-multipart to read form bodies, which is how Twilio sends webhook data.

pip install fastapi uvicorn python-multipart twilio
export TWILIO_AUTH_TOKEN="your-auth-token"
export HUMAN_AGENT_NUMBER="+15551234567"

In the Twilio Console, set the phone number's "A call comes in" webhook to https://your-domain/voice/incoming with method POST. For local testing, expose your server with a tunnel such as ngrok and use that HTTPS URL.

How do you write the FastAPI webhook?

The FastAPI app below handles the full conversation loop. It validates every request, keeps a short history per call keyed by CallSid, and hands off to a human on request.

import os

from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import Response
from twilio.request_validator import RequestValidator
from twilio.twiml.voice_response import VoiceResponse

app = FastAPI()
validator = RequestValidator(os.environ["TWILIO_AUTH_TOKEN"])
calls: dict[str, list[dict]] = {}  # CallSid -> conversation history
MAX_MESSAGES = 10


def generate_reply(history: list[dict]) -> str:
    """Plug your LLM in here.

    `history` is a list of {"role": "user" | "assistant", "content": str}.
    Call your LLM provider with a system prompt plus this history and
    return the reply text. Keep replies to one or two short sentences.
    """
    raise NotImplementedError("Connect your LLM here")


async def twilio_form(request: Request) -> dict:
    form = dict(await request.form())
    signature = request.headers.get("X-Twilio-Signature", "")
    if not validator.validate(str(request.url), form, signature):
        raise HTTPException(status_code=403, detail="Invalid Twilio signature")
    return form


def speak_and_listen(text: str) -> Response:
    twiml = VoiceResponse()
    gather = twiml.gather(
        input="speech",
        action="/voice/turn",
        method="POST",
        speech_timeout="auto",
    )
    gather.say(text)
    twiml.say("Sorry, I did not hear anything. Goodbye.")
    return Response(content=str(twiml), media_type="application/xml")


@app.post("/voice/incoming")
async def incoming(request: Request) -> Response:
    form = await twilio_form(request)
    calls[form["CallSid"]] = []
    return speak_and_listen("Hi, thanks for calling. How can I help you today?")


@app.post("/voice/turn")
async def turn(request: Request) -> Response:
    form = await twilio_form(request)
    history = calls.setdefault(form["CallSid"], [])
    speech = form.get("SpeechResult", "")

    if "human" in speech.lower() or "agent" in speech.lower():
        twiml = VoiceResponse()
        twiml.say("Sure, connecting you to a person now.")
        twiml.dial(os.environ["HUMAN_AGENT_NUMBER"])
        calls.pop(form["CallSid"], None)
        return Response(content=str(twiml), media_type="application/xml")

    history.append({"role": "user", "content": speech})
    reply = generate_reply(history[-MAX_MESSAGES:])
    history.append({"role": "assistant", "content": reply})
    return speak_and_listen(reply)

Run it with uvicorn main:app --host 0.0.0.0 --port 8000. The generate_reply function is the only place your LLM plugs in. Put your system prompt, retrieval step and provider call there, and return plain text.

Why the history is trimmed

Each turn sends only the last MAX_MESSAGES messages to the LLM. Phone conversations are short, and a smaller history keeps prompts fast and cheap. The calls dictionary lives in memory, so it only works with a single worker process. For multiple workers or servers, store history in a shared store such as Redis, keyed by CallSid.

How do you keep latency low?

Latency is the biggest quality problem in voice agents. The caller hears silence while Twilio transcribes speech, your server calls the LLM, and Twilio converts the reply to speech. Every second of delay feels long on the phone.

Keep the LLM's replies to one or two sentences, and say so in the system prompt. Choose a model that responds quickly over one that writes long, detailed answers. Host the FastAPI server in a region close to Twilio and to your LLM provider, and avoid slow work such as large retrieval calls in the request path.

For even lower latency, Twilio Media Streams sends raw call audio over a WebSocket so you can stream speech recognition and speech output yourself. Media Streams is more complex, so start with <Gather> and move only if the delay is a real problem.

How do barge-in and handoff to a human work?

Barge-in means the caller can interrupt the agent while it is speaking. Because <Say> is nested inside <Gather>, Twilio listens during playback and stops the speech when the caller starts talking. The bargeIn attribute on <Gather> controls this behavior.

Handoff to a human uses the <Dial> verb, which connects the current call to another phone number. The example above triggers handoff on a simple keyword check. In a real agent, let the LLM decide by returning a flag or a tool call, and also hand off after repeated failed turns so callers are never stuck in a loop.

How do you secure a Twilio webhook?

Your webhook URL is public, so anyone could post fake call data to it. Twilio signs every request with your auth token and sends the signature in the X-Twilio-Signature header. twilio.request_validator.RequestValidator recomputes that signature from the URL and form fields and rejects mismatches.

Validation fails if the URL your app sees differs from the URL Twilio called. This happens behind a reverse proxy such as Nginx, where the app may see http instead of https. Run Uvicorn with --proxy-headers and make sure the proxy forwards the original scheme and host.

Summary

An AI phone agent with Twilio and FastAPI is a webhook loop: <Gather input="speech"> collects the caller's words, your generate_reply function asks the LLM, and <Say> speaks the answer. Keep replies short for low latency, support barge-in and <Dial> handoff, and validate every request signature. To design the prompts, see the prompt engineering checklist, and for a production voice agent built for your business, see the AI applications service.