VoiceCraft is a voice AI platform for small and medium businesses. You build an AI phone agent in a dashboard, point a phone number at it, and it answers calls: booking appointments, checking availability, taking messages, sending a follow-up text. Simple to describe, and deceptively hard to build, because a phone call is the most unforgiving UI I have ever shipped against.
A web app can show a spinner. A chatbot can stream tokens while you wait. A phone call has neither. If the agent pauses for a beat too long, the caller talks over it. If it talks over the caller, it feels rude and broken. There is no undo, no loading state, and no second chance at a first impression. Every problem below comes back to the same root cause: a conversation happens in real time, and humans notice delays of a few hundred milliseconds.
Here are the parts that were genuinely hard.
The latency budget of a phone call
The pipeline is conceptually a loop: speech to text (STT), then a language model decides what to say (LLM), then text to speech (TTS), then audio back to the caller. Run end to end, the naive version feels like talking to someone on a bad satellite link.
The uncomfortable truth is that most of the latency is not yours to optimize. The model’s “thinking” time and the TTS engine’s synthesis time are external calls measured in hundreds of milliseconds each, and you do not get to make them faster. So the engineering is not about shrinking those costs. It is about hiding them.
A few techniques that moved the needle, taking caller-perceived “first response” latency down by roughly half:
- Do the setup work while the phone is still ringing. Loading the agent’s configuration, looking up the caller, and warming the knowledge index all happen concurrently during the ring window, before anyone says hello. By the time the call connects, the agent is already armed.
- Preemptive synthesis. Start generating the spoken response before the model has fully finished its turn, so TTS time-to-first-byte overlaps with the model’s tail instead of running after it. You occasionally throw away audio if the caller interrupts, but the average turn gets noticeably snappier.
- Pre-synthesized filler phrases. When the agent has to call a tool (“let me check our availability”), a short, pre-rendered filler plays instantly from cached audio while the real work happens in the background. Zero synthesis latency on the part the caller actually hears first.
- Turn off the model’s deep thinking for receptionist turns. Modern models default to a chunk of server-side reasoning that adds one to two seconds before the first token. On a phone, one to two seconds is the difference between a receptionist and a hold message. For a “what time works for you?” turn, that reasoning buys nothing and costs everything. Setting thinking to minimal was one of the biggest wins.
- Pre-warm what you can, honestly document what you can’t. Voice-activity-detection weights load into idle workers so the first call does not eat a cold start. Some things genuinely cannot be pre-warmed, and pretending otherwise just hides the latency somewhere worse.
Barge-in: letting the caller interrupt
This is the one I learned the most from.
Real conversations are full of interruptions. You start to say “our hours are nine to-” and the caller cuts in with “do you take walk-ins?” A good phone agent has to stop talking the instant the caller starts. This is called barge-in, and getting it wrong makes the agent feel like a hold message.
I shipped barge-in, it worked, and then one day callers could not interrupt the agent at all. The agent would monologue straight through them. The bug taught me something subtle about how these systems fit together.
The correct design turned out to be pause first, decide later:
The interruption logic had a “minimum words” gate: only treat the caller as interrupting once they have said at least a couple of words, so a cough or a stray “um” does not chop off the agent mid-sentence. Reasonable on paper. The problem is when that gate is checked. Voice-activity detection fires the moment it hears speech - within milliseconds. But the word count comes from the STT transcript, and I had moved to a streaming STT whose interim transcript lagged by close to a second. So at the exact instant the gate was evaluated, the transcript was still empty. Zero words. Below the threshold. Interruption silently suppressed. Every time.
The fix was to stop trying to count words at the wrong moment and instead use pause-and-resume:
- The moment voice activity exceeds a short minimum duration, pause the agent’s playback immediately. No waiting on a transcript.
- If a real transcript materializes within a couple of seconds, it was a genuine interruption - hand the turn to the caller.
- If nothing materializes (it was a cough, background noise, a passing truck), resume the agent right where it left off.
Same noise-robustness goal, but it no longer depends on a transcript existing at a moment when it cannot possibly exist yet. The lesson generalizes: in a real-time pipeline, two signals you treat as simultaneous (sound detected, words recognized) can be a full second apart, and any logic that assumes they line up will fail in ways that are maddening to reproduce.
A related gotcha: the framework disabled interruptions for the first few seconds of the opening greeting, to let client-side echo cancellation calibrate. On a phone call the upstream carrier already handles echo, so that warmup was doing nothing except preventing callers from interrupting the greeting - the single most common place a caller wants to jump in. Turning it off made the agent feel alive from the first word.
Knowing when the caller is done speaking
The flip side of interruption is endpointing: deciding the caller has finished a turn so the agent can respond. Cut in too early and you trample the caller. Wait too long and the agent feels frozen.
Silence is the obvious signal, and the tempting one - it needs no model and fires instantly. It is also a terrible signal, because people pause mid-thought (“I’d like to book for, uh… Thursday”). I use voice-activity detection for raw speech boundaries but let a small machine-learning turn detector make the actual “are they done?” call, with a floor and a ceiling on how long it will wait.
That ceiling matters. Early on it was set generously, and production logs showed turns where the detector waited the full timeout on a long pause - five seconds of dead air, which on a phone feels like the call dropped. Lowering the ceiling, informed by real call data rather than a guess, fixed the “is it still there?” feeling.
There was also a sharp dependency between STT choice and turn-taking. One STT model’s final transcripts routinely arrived later than the agent’s timeout for waiting on them, so the turn boundary never resolved and the agent simply never replied. The fix was a different, streaming-optimized STT model. It is a good reminder that in a pipeline, a component’s tail latency, not its average, is what breaks the experience.
Finally, for genuine silence (nobody says anything for a while), the agent escalates gently, then directly, and after a couple of tries it stops asking the model what to say and falls back to a fixed, localized goodbye and hangs up cleanly. In long multi-turn calls the model tends to drift off its wrap-up instructions, a well-documented failure mode as conversations grow longer, so for the “wrap this up safely” path I deliberately took the model out of the loop and made it deterministic.
Don’t let one bad webhook hang up the call
Every external dependency in this system can fail: an STT provider, the model, the TTS engine, or the business’s own booking backend. In a web app a failed request shows an error toast. On a live call, a hang means dead air, and dead air means the caller hangs up.
So resilience is designed in at every leg:
- Provider routing with fallbacks. STT, LLM, and TTS are each abstracted behind a small registry, and each has a fallback so a primary outage degrades to a secondary instead of failing the call. Adding or swapping a provider is a config change, not a redeploy.
- A real subtlety in TTS failover: a voice is not portable across providers. A voice name that is valid on one engine is meaningless on another, so the fallback has to pick a gender-matched voice that is actually valid on the target engine, or the “recovery” just produces a second error. Resilience code that itself throws is worse than no resilience code.
- Per-tool timeouts and circuit breakers. Every tool call (check availability, book, send a message) is wrapped in a hard timeout and a circuit breaker scoped per agent and per tool. If one business’s booking webhook starts timing out, the breaker opens and the agent gracefully says “I’ll have someone follow up with you” instead of leaving the caller in silence - and one tenant’s broken integration can never wedge anyone else’s calls.
The guiding principle: the call must always make forward progress. Every failure path resolves to the agent saying something sensible, never to silence.
Grounding tools in real data
An agent that confidently invents an appointment slot is worse than useless. Two things keep tool use honest.
First, tools never throw. A failed tool returns a friendly sentence the agent can speak, never an exception that crashes the turn. The tool definitions are heavily prompt-engineered against misuse: only book after availability has been checked and the caller has explicitly confirmed, never quietly substitute “take a message” for “book an appointment,” always resolve a relative date like “next Tuesday” to a concrete date before calling.
Second, booking is idempotent. Models sometimes call a tool twice. An in-flight de-duplication plus a short result cache, keyed on the specifics of the request, means a double-call can never become a double-booking.
My favorite bug here was about the order of operations. When a caller asked for an evening appointment, they were never offered one. The cause: the code capped the results to a handful of slots (humans can only hold three or four options in their head) and then tried to filter by time of day. But the earliest slots are in the morning, so the morning slots always won the cap and the evening filter had nothing left to work with. The fix was a one-line reordering - filter to the requested time window first, then cap. The bug was not in either step; it was in their sequence.
The agent also pulls in business-specific knowledge per turn, with a relevance floor so low-confidence matches are dropped rather than spoken as fact, and turns that find nothing useful are logged as knowledge gaps to improve later. If the knowledge index does not match the agent’s embedding space, it is refused outright rather than served, because serving uncorrelated text the agent would then cite as fact is the worst possible failure for a system whose whole job is to be trustworthy on the phone.
Telling people they are talking to a robot
If a machine sounds human enough, you have an ethical and increasingly a legal obligation to disclose that it is a machine. California’s Bot Disclosure law and the EU AI Act both point in the same direction.
So disclosure is on by default, spoken at the very start of the call, and localized. Two engineering decisions made it correct rather than cosmetic:
- The disclosure is spoken verbatim, not generated. If you ask the model to “greet the caller and mention you’re an AI,” it will sometimes rephrase the notice into something vague or drop it entirely. A legally required statement cannot be left to a model’s discretion, so it is played as a fixed line before the model ever takes a turn.
- It has to sound like a person saying it. A flat, robotic meta-statement was actually rejected by a TTS engine’s own safety classifier, which failed the call over to a backup voice. The fix was to reword the disclosure as natural conversational dialogue - something a friendly receptionist would actually say - which both satisfied the classifier and, ironically, made the required disclosure feel less robotic. Compliance and good UX turned out to be the same edit.
A war story: thinking models will eat your output budget
A quick one, because it bit me twice in different places. After each call, the agent generates a short summary. One day the summaries came back empty. No error, no exception, just blank.
The cause: the summary used a “thinking” model, and that model spent its entire output token budget on internal reasoning tokens, leaving nothing for the actual answer. The response came back finished, with a “hit the token limit” reason and an empty body. The same failure mode had shown up earlier when long tool-narration responses silently truncated.
Two lessons. First, a thinking model can silently spend its entire output budget on reasoning - the same reason receptionist turns run on minimal thinking, except here it cost a result instead of latency. Second, never let a failure be silent: empty outputs are now logged loudly, because “it returned nothing and told no one” is the hardest kind of bug to chase.
Sounding human in many languages
One last detail I am proud of, because it is rarely discussed. Translating the agent’s words is the easy part. Sounding natural is the hard part, and it is different in every language.
Many languages have a formal, literary register (how the news is read) and an everyday spoken register (how people actually talk), and a receptionist must use the latter. Many also have a respectful second-person form you are expected to use with an adult stranger. So beyond translation, the agent carries per-language guidance on register and politeness, so it sounds like a real local receptionist rather than a textbook. A voice agent that is grammatically perfect but socially wrong still sounds like a robot.
Takeaways
If I had to compress all of this into a few transferable lessons:
- In a real-time pipeline, signals you assume are simultaneous can be a second apart. The barge-in bug, the endpointing stalls, and the empty summaries were all timing or ordering problems, not logic errors in any single component.
- You cannot make the slow parts fast, so hide them. Almost every latency win was about overlapping work or doing it earlier, not about making any one call quicker.
- Every failure path must resolve to the agent saying something. Silence is the one unacceptable output, so design every timeout, fallback, and circuit breaker to end in speech.
- Doing the responsible thing and the polished thing are often the same edit. Making the AI disclosure sound human satisfied both a safety classifier and the caller.