AI services · Voice

How Voice AI Actually Works, and Which Kind You Need

Five stages turn text into a voice, and the one everybody judges is not the one that breaks. Here is the pipeline, the failure that causes most complaints, and the latency arithmetic that decides whether a voice agent feels like a conversation or a phone tree.

“Voice AI” is three different products

They get sold under one heading and they solve unrelated problems.

What it doesDirectionYou need it for
Speech synthesisText goes in, audio comes outNarration, announcements, anything read aloud
Speech recognitionAudio goes in, text comes outTranscription, captions, voice commands
Voice conversionAudio in, different-sounding audio outDubbing, changing a speaker’s voice, accent work

A conversational agent needs all three layers plus a language model between them, which is why agents are harder than any single piece suggests. Work out which of the three you are buying before comparing anything.

How text becomes a voice: five stages

Nearly every synthesis system works through the same sequence, and knowing it tells you where to look when output is wrong.

Text normalisation

Written text is converted to what a person would actually say. “Dr.” becomes doctor or drive. “1996” becomes nineteen ninety-six, but “19.96” becomes nineteen point nine six. Currency, dates, units, abbreviations and acronyms all get expanded here.

Grapheme to phoneme

Words become sounds. Mostly rules and a pronunciation dictionary, with a model guessing at anything unknown. Names, loanwords and technical terms live or die at this step.

Prosody prediction

Where to put stress, how long each sound lasts, where pitch rises and falls, where to breathe. This is the stage that decides whether output sounds like a person or a reading machine.

Acoustic modelling

Phonemes plus prosody plus a speaker identity become a spectrogram, a picture of the sound across frequency and time.

Vocoding

The spectrogram becomes an actual waveform you can play. Modern neural vocoders are the reason output stopped sounding metallic.

Older systems stitched together recorded fragments of a real speaker, which is why they sounded fine on expected phrases and strange on anything else. Current systems generate the audio, which is why they handle unfamiliar text smoothly and occasionally invent something confidently wrong.

Where quality actually dies

People judge a voice on timbre, the quality of the sound. Almost no complaints are actually about timbre.

The failures that make listeners wince nearly all come from stage one, which is the least glamorous part of the system and the part no demo stresses.

  • A street abbreviation read as a job title, or the reverse.
  • A version number read as a decimal, or a decimal read as a date.
  • An acronym spelled out when it should be said as a word, or said as a word when it should be spelled.
  • Currency in the wrong order, or a unit expanded into the wrong word.
  • A customer’s name mangled, which in a service context is worse than any other error on this list.

This matters when choosing, because normalisation quality is almost never demonstrated. Demos use clean prose. Your content has product codes, prices, dates, abbreviations and names.

So test it with your ugliest real text, not a paragraph you wrote for the test. Most of the difference between systems shows up there rather than in how pleasant the voice sounds.

Why a cloned voice sounds right for a sentence and wrong for a paragraph

Cloning works by reducing a voice to a speaker embedding: a vector capturing what makes that voice identifiable, separate from the words and separate from the delivery. Give the system the embedding plus new text and it produces that voice saying something it never said.

Identity transfers well. A few seconds of clean audio can be enough to be recognisable, and more material mainly improves robustness rather than likeness.

What does not transfer is intent. The embedding carries who is speaking, not what they mean. Prosody is predicted from the text alone, so the system has no idea which word you would have emphasised, where you would have slowed down for effect, or that this sentence is the punchline of the previous one.

That is why clones pass on a single line and feel subtly wrong across a paragraph. The voice is right and the performance is generic. For narration it is usually acceptable. For anything emotional it is the thing that gives it away, and no amount of extra training audio fixes it, because the limitation is in what prosody prediction can see.

The latency arithmetic nobody does

If you are building anything conversational, this section matters more than voice quality, and it is where most projects misspend their effort.

A person speaks. Before they hear anything back, six things happen in sequence. The user hears silence for the sum of all of them.

StageNaive buildTypical first buildTuned, streamingAggressive
Deciding you stopped talking700500250150
Finalising recognition40025012080
Model’s first token900600350250
Synthesis to first audio500300150100
Network round trip1501007040
Playback buffer120805040
Total silence, ms2,7701,830990660

Those are illustrative budgets rather than measurements of any product, but the proportions hold across builds. For context: in natural conversation people start replying within roughly a couple of hundred milliseconds, and a gap approaching a second reads as hesitation. Past that, callers start talking over the system.

Now look at where the time sits in the tuned column.

StageShare of total delay
Model’s first token35%
Deciding you stopped talking25%
Speech synthesis15%
Recognition12%
Network7%
Playback buffer5%

The voice model is fifteen per cent of the problem. Make synthesis instantaneous, free, and perfect, and the system gets 15% faster. Teams spend months choosing it.

Meanwhile the largest single chunk after the language model is deciding the user has finished speaking, a piece of configuration most projects never touch. Wait too long and the system feels slow. Cut it too short and it interrupts people mid-sentence, which is worse. That trade-off is tuned, not bought.

The practical consequence: if a voice agent feels sluggish, measure each stage before changing the voice. In most builds the fix is streaming, endpoint tuning, or moving services closer together, not a different synthesis provider.

Streaming is the feature that matters

One capability separates systems that work in conversation from those that do not: whether audio starts playing before the whole sentence has been generated.

A batch system takes the full text, generates the full clip, then plays it. The user waits for all of it. A streaming system emits audio as it goes, so the measure that matters is time to first audio, not time to finish.

The related number is the real-time factor: seconds of compute per second of audio produced.

Real-time factorSpeedSuitable for live conversation
0.110× real timeComfortably
0.33.3× real timeYes
0.52× real timeYes, with little headroom
1.01× real timeNo, it falls behind under load

For narration none of this matters; you can wait. For anything interactive it decides whether the product works at all, and it is the first question to ask.

Which kind of tool suits which job

Described by capability rather than by product, because the right answer changes with the job and not with the brand.

If you are doingPrioritiseIgnore
Narration, audiobooks, coursesProsody quality, pronunciation control, long-form consistencyLatency entirely
Live agent or phone systemStreaming, time to first audio, interruption handlingSubtle expressiveness
Accessibility, screen readingClarity, speed control, correct normalisationEmotional range
Dubbing or localisationVoice conversion, timing to fit the originalGeneric voice libraries
Brand voice at volumeCloning stability, pronunciation dictionaries, licensing termsBreadth of stock voices
Internal prototypesSimple pricing, easy APIAlmost everything else

Two capabilities are worth paying for in almost every serious deployment, and both are easy to overlook:

  • A pronunciation dictionary. Somewhere you can tell the system, permanently, how your product and company names are said. Without it you are fighting the same error forever.
  • Markup control over delivery. The ability to force a pause, change emphasis, spell something out or set a speaking rate. Without it, fixing a bad reading means rewriting the text until the model happens to get it right.

Consent, and who owns a voice

This is the part with real legal exposure, and it is short because the rules are simple.

A voice is identifying, like a face. Cloning one without permission creates problems regardless of how good the output is, and several jurisdictions have been actively legislating on synthetic likeness.

  • Get written consent naming the specific uses. Permission to record a product video is not permission to clone the voice for anything else.
  • Agree what happens when the person leaves. A former employee’s cloned voice still answering your phones is a dispute waiting to happen.
  • Check whether the provider trains on your uploads. If the voice you cloned can reappear elsewhere, that is a problem you promised someone would not occur.
  • Disclose synthetic voices where people would reasonably expect a human. Rules on this are tightening, and it is becoming an obligation rather than a courtesy.

What to test before you commit

Feed it your worst text

Product codes, prices, dates, abbreviations, customer names. Not a clean paragraph. This is where systems separate.

Measure time to first audio

Not total generation time. If the tool cannot stream, it cannot hold a conversation.

Listen to a long passage

Five minutes, not five seconds. Prosody problems only appear at length, and so does listener fatigue.

Try to correct a mistake

Find the deliberate mispronunciation and fix it. If the only route is rewording the text, you will be doing that forever.

Questions readers ask

How much audio is needed to clone a voice?

Less than people expect for recognisability, and more than people expect for reliability. Short samples give a voice that sounds right in easy conditions and degrades on unusual words or long passages.

Why does it mispronounce our product name?

Because the name is not in the pronunciation dictionary and the model is guessing from spelling. The fix is a dictionary entry or phonetic markup, not a better voice.

Can listeners tell it is synthetic?

Increasingly not from a single sentence. Over several minutes most listeners sense something, usually from flat prosody rather than from the sound itself.

Is cloud or on-device better?

Cloud gives better quality and easier scaling. On-device gives predictable latency and keeps audio local, which matters where recordings cannot leave. Pick on the constraint, not the quality.

Our agent feels slow. Should we change the voice provider?

Measure first. Synthesis is usually a small share of the delay. The common culprits are non-streaming components, over-cautious endpointing and services in different regions.

Do we have to say it is AI?

Increasingly yes in customer-facing contexts, and sensible everywhere. Disclosure costs little and the alternative ages badly.

The short version

Five stages: normalise, convert to sounds, predict delivery, model the audio, make the waveform. Most quality complaints come from the first stage, which no demo stresses, so test with your ugliest real text. Cloning transfers identity but not intent, which is why clones pass on one line and feel flat over a paragraph. For anything conversational, synthesis is about fifteen per cent of the delay, so measure the whole chain before blaming the voice. And get consent in writing, naming the uses.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top