How to Train Your Own TTS Model Locally with Pocket TTS
Kyutai open-sourced the full Pocket TTS training stack. Here's how to train a custom CPU-runnable voice model on your own GPU and data.
What is Pocket TTS and why does its training release matter?
Pocket TTS is a small text-to-speech model from Kyutai built to run inference on a CPU, no GPU required once the model is trained. Kyutai previously released the model itself for people to run locally. What changed recently is that Kyutai open-sourced the entire training stack behind it: the data preparation pipeline, the forced-alignment tooling, the training recipes, and the evaluation scripts. That means anyone with a GPU can now train a custom version of this model from scratch, in a different language, on a different voice, or on entirely private data, and then deploy the result to run on ordinary CPU hardware.
This matters because most TTS training pipelines are either closed, poorly documented, or require you to stitch together several unrelated research repos. Kyutai’s release packages the whole loop, data to teacher model to distilled student model, into one repository with working scripts.
TL;DR
- Kyutai released the full training stack for Pocket TTS, not just the model weights, covering data prep, forced alignment, training, and distillation.
- The pipeline runs on a single GPU (an H100 was used in testing, but the workflow doesn’t demand one) and installs cleanly with the uv Python package manager in one command.
- Training happens in two phases: a 24-layer teacher model is trained on the GPU, then distilled into a 6-layer student model that is small enough to run inference on CPU.
- Quality shifts happen at rough checkpoints: babbling turns into recognizable words by around 15,000 steps, the model reads arbitrary text by 50,000 steps, and the voice stops sounding synthetic by roughly 200,000 steps.
- For your own data, Kyutai recommends at least 100 hours of paired audio and transcript for a usable model, and 1,000+ hours if you want production-grade output.
- The default example trains on a reproducible slice of LibriVox audiobook data via the LibriTTS/LibriVox pipeline, using pre-computed word alignments pulled from Kyutai’s own alignment dataset rather than computing them yourself.
- Full training to 200k steps on a single H100-class GPU takes on the order of a day or more, which is modest compared to typical from-scratch speech model training runs.
Other agents ship a demo. Remy ships an app.
Real backend. Real database. Real auth. Real plumbing. Remy has it all.
How does the training pipeline actually work?
The process breaks down into four stages.
1. Prepare data. You need speech audio paired with exact transcripts. Kyutai’s example script downloads a slice of LibriVox audiobook recordings (read by volunteers) rather than the entire ~31,700-hour corpus, grabbing a reproducible subset instead. It doesn’t cut or re-encode audio: it downloads one MP3 per book chapter and simply records where each sentence starts and how long it runs. Dead links get skipped and counted rather than crashing the run, and the whole download step is resumable if interrupted.
2. Align. The model needs to know exactly when each word was spoken inside the audio, not just what the words are. This is done with forced alignment, mapping words in the transcript to timestamps in the waveform. In the default example, these alignments come pre-computed from Kyutai’s own alignment dataset rather than being generated locally, which saves a compute-heavy step.
3. Train the teacher. This is the heavy lifting: a 24-layer model trained from scratch on the GPU. Training quality moves through visible phases as steps increase, from meaningless babble early on to full readable speech by the mid-training mark, to a voice that sounds natural by the later stage.
4. Distill the student. The large teacher model isn’t what ends up running on your CPU. It gets distilled into a much smaller 6-layer student model, which is the version actually deployed for fast CPU inference. This distillation step is what makes the end product practical to run without a GPU at inference time.
The output of the data-prep stage is a pair of JSONL manifest files: one raw manifest with audio paths, start times, and durations, and one “aligned” version that adds word-level timing data. A small held-out validation set (a few hundred utterances from different books and speakers) is kept separate from training data specifically to catch memorization rather than genuine learning.
What do you need to train on your own voice or language?
The core requirement is simple: audio recordings paired with accurate transcripts. Beyond that, three things determine whether the result is usable.
Quantity. Kyutai’s guidance points to roughly 100 hours of paired audio and transcript as a floor for something that sounds decent, and 1,000+ hours if you’re aiming for production quality.
Quality. Clean audio and accurate transcripts matter more than raw volume. Noisy recordings or sloppy transcription produce a bad model regardless of how much data you feed it.
Format. You place your audio files and matching transcript files into the repo’s data directory structure (audio in one folder, transcript text in another). The exact folder layout is flexible as long as the paths in your manifest match where the files actually live. The one thing the released repo doesn’t include out of the box is a script to convert raw audio and transcript files into the required JSONL manifest format, so that conversion step has to be handled separately before feeding data into the existing prepare_data script.
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
Once your JSONL manifest exists in the expected location, you run the same data preparation script used for the default LibriVox example, and it builds out the rest of the pipeline against your custom data.
How do you actually run the training job?
After environment setup (the repo installs cleanly with a single uv sync command, pulling in both training and evaluation dependencies without needing a separate environment) and after data preparation finishes, training is kicked off with one command pointed at a configuration file. That config file controls everything: which dataset to use, the number of layers (24 for the teacher), file paths for your prepared manifests, and optimizer settings like batch size, step count, logging frequency, and checkpoint intervals.
For most users, the defaults for optimizer and checkpoint settings can be left alone. The main things you’re likely to change are the dataset name/language label and the paths to your own JSONL files.
Hardware-wise, the training run used an H100 GPU in testing, but the workflow does not require that specific card. Any reasonably modern GPU can handle it, and GPUs can be rented by the hour from cloud compute providers rather than purchased outright, which lowers the barrier for a one-off training run considerably.
Is training your own TTS model locally worth it?
For most people who just want a working CPU-runnable TTS voice, using Kyutai’s already-released Pocket TTS model directly is faster and requires no GPU at all. Training your own model from scratch is worth it specifically when you need a voice or language the base model doesn’t cover well, or when you have a private dataset (a specific speaker, a specific accent, a specific domain vocabulary) that you want baked into the model rather than prompted around.
The tradeoffs are real: you need a meaningful amount of clean paired audio and transcript data, a GPU for a day or more of compute, and patience through a multi-stage pipeline where quality only becomes apparent gradually over tens of thousands of training steps. But because the entire stack, from data download to teacher training to CPU-ready distillation, is now openly available and scriptable, the technical barrier to trying it has dropped substantially compared to assembling a custom TTS pipeline from separate research projects.
Frequently Asked Questions
What hardware do I need to train a Pocket TTS model?
A single GPU is sufficient. Testing was done on an H100, but the pipeline doesn’t require that specific card, and commodity or rented cloud GPUs work for the training run.
How much audio data do I need for a custom voice or language?
Kyutai’s guidance suggests at least 100 hours of paired audio and transcript for a usable result, and 1,000+ hours for production-quality output. Clean audio and accurate transcripts matter as much as quantity.
How long does training take?
On a single high-end GPU, reaching the full 200,000-step training target takes roughly a day or more. Recognizable words start appearing around 15,000 steps, and the model reads arbitrary text by around 50,000 steps.
Do I need to compute word alignments myself?
Not necessarily. The default pipeline pulls pre-computed word-level alignments from Kyutai’s own alignment dataset rather than requiring you to run forced alignment yourself, though custom datasets need their own alignment step.
Why train a big 24-layer model if the final product is smaller?
The 24-layer model acts as a teacher. Its knowledge gets distilled into a compact 6-layer student model, which is the version actually deployed. The larger model produces better training signal; the smaller one is what makes CPU-only inference practical.