Part 3: Speech Models Converge Toward LLMs — TTS and Speech2Speech Training Data

Speech Models: “Text-to-Speech” and “Speech-to-Speech”

The audio-family models we cover this time come in two broad kinds. One is text-to-speech (TTS), a model where text goes in and read-aloud audio comes out. Since the input is text and the output is audio, you could also write it as “TXT2Speech” — it’s the technology that reads sentences aloud in car navigation systems and smart speakers.

The other is Speech2Speech (S2S), a model where audio goes in and different audio comes out. Speech translation (Japanese audio → English audio) and voice conversion (one person’s voice → another’s) fall here.

Just like the LLM (TXT2TXT) and image generation (TXT2IMG) from earlier parts, the audio family can be captured as “put in input X, get output Y.” What’s different is that the objects handled are “sound.” So how do you make the model learn that correspondence? We’ll look at it starting with TTS.

Audio Training Data Starts from “Text-Audio Pairs”

Training data for text-to-speech (TTS) is basically pairs of text and the audio that reads it aloud. A typical dataset looks like this:

dataset/
    speaker001/
        0001.wav
        0002.wav
        metadata.csv

0001.wav|It's nice weather today.
0002.wav|It will probably rain tomorrow.

As long as “which text corresponds to which audio” is aligned, you can train. So far this is the same as LLMs and images — it’s about building “input” and “answer” pairs. The difference is that the answer is “sound.” How you represent that sound splits TTS into two broad generations.

The Classic Approach — the Answer Is a Mel Spectrogram

First, handling audio as a raw waveform carries too much information. So many models convert it to a Mel spectrogram (a frequency representation tuned to human hearing).

import librosa

wave, sr = librosa.load("0001.wav", sr=22050)

mel = librosa.feature.melspectrogram(
    y=wave, sr=sr, n_fft=1024, hop_length=256, n_mels=80,
)

The text side is tokenized. Then the training data is:

input:  text (token sequence)
answer: Mel spectrogram

For a classic model like Tacotron2, training is almost just this:

predicted_mel = model(tokens)
loss = F.l1_loss(predicted_mel, target_mel)

Finally, you need to turn the Mel spectrogram back into an actual waveform. That’s the job of a vocoder — Griffin-Lim in the old days, now HiFi-GAN, BigVGAN, UnivNet, and so on.

Text ─▶ TTS Model ─▶ Mel ─▶ Vocoder ─▶ audio

The Recent Approach — the Answer Is an “Audio Token”

After Tacotron, TTS didn’t settle on a single approach. As ideas, they divide into the autoregressive (audio-token) type, which turns audio into discrete tokens and predicts the next token like an LLM (VALL-E and others), and the diffusion / flow-matching type, which — just like image generation in Part 2 — adds noise to an audio representation and learns to restore it (Voicebox and others).

But real models don’t split cleanly into these two; in practice, hybrids that combine both are common. CosyVoice, for instance, pairs an LLM part that autoregressively generates audio tokens from text with a flow-matching part that turns those tokens into audio. Seed-TTS has both an autoregressive version and a diffusion (DiT) version; F5-TTS is a non-autoregressive flow-matching model; XTTS combines autoregressive token generation with a vocoder. Real systems are assemblies of multiple approaches.

Here we focus on the autoregressive (audio-token) part, where the series’ “each field converging toward the LLM’s shape” trend is clearest. In that view, the model predicts audio tokens instead of Mel spectrograms.

The method closely resembles the image latent: convert the audio into discrete tokens with a neural codec like Encodec.

tokens = encodec.encode(wave)
# e.g. [[54, 91, 12, ...], [83, 19, 42, ...], ...]

Then the training data becomes:

input:  It's nice weather today (text tokens)
answer: 54 91 12 ... / 83 19 42 ... (audio tokens)

The training code is now completely an LLM’s.

text_ids = tokenizer(text)
target_audio = encodec.encode(wave)

logits = model(text_ids)

loss = F.cross_entropy(
    logits.reshape(-1, vocab_size),
    target_audio.reshape(-1),
)

Part 1’s LLM predicts “text tokens”; this TTS predicts “audio tokens.” Only the contents of the token changed, from letters to sound — the skeleton of learning is the same. The point from earlier parts, that the fields of generative AI are converging to the same shape, holds for audio too.

Furthermore, the latest models feed in not only text but speaker ID, emotion, speaking rate, pitch, language, accent, and even a reference audio as conditions, each turned into an embedding. “Who speaks, with what emotion, at what speed” is included in the training data.

Speech2Speech — Audio-to-Audio Conversion

Audio also has tasks that convert audio directly to audio without going through text. This is Speech2Speech (S2S), which is in fact a fairly broad concept.

• Speech translation (Japanese → English)
• Voice conversion (male → female)
• Denoising
• Emotion conversion, speaker conversion
• Real-time conversational AI

For all of them, the training-data skeleton is “input audio → output audio.” For denoising, for instance, you prepare the “noisy” and “clean” versions of the same utterance.

noisy = load_audio("noisy.wav")
clean = load_audio("clean.wav")

pred = model(noisy)
loss = F.l1_loss(pred, clean)

For voice conversion you gather many “male audio → female audio” pairs; for speech translation, “Japanese audio → English audio” pairs.

And recently, S2S is dominated by the token approach too. You convert both input and output audio into audio tokens and learn that conversion.

src_tokens = encodec.encode(source_wave)
tgt_tokens = encodec.encode(target_wave)

logits = model(src_tokens)

loss = F.cross_entropy(
    logits.reshape(-1, vocab),
    tgt_tokens.reshape(-1),
)

“Input token sequence → output token sequence” — again, almost the same structure as an LLM learning translation.

End-to-End Conversational AI — No Text in the Middle

One step further is real-time conversational AI (a setup like OpenAI’s Advanced Voice Mode). Here, audio is processed without converting to text.

audio ─▶ Audio Encoder ─▶ Audio Token ─▶ LLM ─▶ Audio Token ─▶ Decoder ─▶ audio

The point is that it never routes through text. This is called Speech-to-Speech End-to-End. At its core is an LLM; only the tokens it handles change from “letters” to “sound.” In other words:

LLM:      text token ─▶ Transformer ─▶ text token
speech AI: audio token ─▶ Transformer ─▶ audio token

Training data for this setup needs more than “input audio → output audio.” You also record metadata like conversation history, speaker ID, emotion, accent, language, and conversational turns, to learn natural exchanges.

Three Side by Side

Lining up the “text and audio” family we’ve covered since Part 1, the convergence is easy to see.

The Difficulty Unique to Audio — Paired Data Is Hard to Gather

Finally, a hurdle specific to audio. S2S is a field where gathering training data is harder than for LLMs or TTS, because you need a correspondence between “input audio” and “ideal output audio.”

• Denoising → you need the “noisy / clean” versions of the same utterance
• Speech translation → you need paired data of the same content spoken in different languages
• Voice conversion → you need pairs of the same content spoken by multiple speakers

Such perfect paired data doesn’t conveniently exist in large volumes. So recently a common approach is to first learn audio representations self-supervised from large amounts of audio, then adapt to the target task with a small amount of paired data. It’s exactly the same idea as the LLM’s “large-scale pretraining + instruction tuning.” Here too, how training data is built comes to resemble other models across the board.

Summary

• TTS training data is “text + audio.” It used to make the Mel spectrogram the answer; recently the answer is audio tokens, making the structure just like an LLM.
• Speech2Speech is “input audio → output audio.” From translation to voice conversion to end-to-end conversational AI, the skeleton converges on token-sequence conversion.
• Because paired audio is hard to gather, the standard is large-scale self-supervised pretraining plus a small amount of adaptation.

Next time, we move to footage that carries both time and space. Video generation, and the world models beyond it — the two look alike but differ completely in what they’re learning for, and in their training data.