PhoneLLM Alpha 1: Pipecat's Purpose-Built Model for Voice Agents
Pipecat's PhoneLLM Alpha 1, a 30B Nemotron fine-tune for phone voice agents, matches GPT-5.6 Terra accuracy at 94% lower cost and lower latency.
What is PhoneLLM Alpha 1?
PhoneLLM Alpha 1 is an open-weight language model built by the Pipecat team at Daily specifically for phone-based voice agents. It’s a full-parameter fine-tune of NVIDIA’s Nemotron 3 Nano 30B-A3B, a hybrid Mamba-Transformer mixture-of-experts model with 30 billion total parameters and 3.5 billion active per token. Instead of chasing general intelligence benchmarks, PhoneLLM optimizes for the three things that actually determine whether a voice agent works in production: fast time-to-first-token, accurate tool calling in multi-turn conversations, and low cost per minute of runtime.
TL;DR
- PhoneLLM Alpha 1 is a full-parameter supervised fine-tune of NVIDIA Nemotron 3 Nano 30B-A3B, trained with the NeMo framework specifically on phone agent conversation data.
- The model uses a mixture-of-experts architecture with only 3.5B active parameters out of 30B total, which keeps inference fast and cheap even at high concurrency.
- On Pipecat’s own PhoneBench v1 benchmark, PhoneLLM matches the accuracy of GPT 5.6 Terra while running about 94% cheaper and with a 1,300ms faster P95 time-to-first-token.
- The model is trained to call tools correctly without needing extended thinking or reasoning tokens enabled, which is where many general-purpose LLMs fail in live voice conversations.
- It’s released under a BSD 2-Clause license with no commercial restrictions, and can be self-hosted on vLLM or SGLang, or deployed directly through Modal’s Auto Endpoints.
- Pipecat’s cost modeling shows self-hosted PhoneLLM can run for roughly $0.00025 per agent-minute on a single NVIDIA B200 at high concurrency.
Why does voice need its own LLM?
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
Most voice agents today are built by bolting a general-purpose frontier LLM onto a speech-to-text and text-to-speech pipeline. That works, but it runs into two structural problems that Pipecat’s team says motivated PhoneLLM.
First, latency. Frontier models are increasingly tuned for use with reasoning or “thinking” tokens turned on, which adds a meaningful delay between a caller’s question and the agent’s spoken response. In a phone call, that delay is felt immediately. People tolerate roughly 1,500ms of voice-to-voice latency in a phone conversation before the interaction starts to feel broken, and that budget has to cover network transport, audio processing, speech-to-text, the LLM’s own generation time, and text-to-speech. Pipecat’s own latency breakdown for a well-optimized agent leaves only about 650ms for the LLM’s time-to-first-token. GPT 5.6 Terra running in its fast mode has a P95 time-to-first-token near 1,900ms, meaning the LLM step alone can blow through the entire latency budget before audio processing is even counted.
Second, reliability of tool use. Voice agents need to actually book the appointment, transfer the call, or look up the account, not just say they did. Pipecat found that many models, including large ones, will confidently claim an action succeeded (“I’ve booked that table for you”) without having invoked the tool at all, especially in longer multi-turn conversations and especially with thinking disabled for speed. That kind of say/do mismatch is a serious failure mode for a customer-facing phone agent, since it isn’t visible to the caller until something goes wrong downstream.
How was PhoneLLM trained and evaluated?
PhoneLLM starts from Nemotron 3 Nano 30B-A3B and goes through full-parameter supervised fine-tuning on the NVIDIA NeMo framework, using conversational data representative of real phone agent traffic (appointment booking, account lookups, customer service escalation, and similar tasks across industries like financial services, healthcare, retail, and hospitality). The model supports a 262,144-token context window and ships in bfloat16 safetensors, and Pipecat recommends running it with temperature set to zero and thinking explicitly disabled, since that’s how it was trained and evaluated.
To measure whether the fine-tune actually helped, Pipecat built a companion benchmark called PhoneBench v1. Voice agent outputs are hard to score automatically: speaking style, factual grounding, and whether a tool call happened at the right moment with the right parameters are subjective in ways that plain string-matching can’t capture. PhoneBench addresses this with a panel of LLM judges, calibrated against human labels, that grade model behavior against high-quality reference conversations across dimensions including telephone speaking style, tool call accuracy, say/do consistency, factual grounding, conversation coherence, authentication and escalation discipline, and overall caller outcome. Crucially, the benchmark’s scenarios, tool lists, and system prompts are kept separate from PhoneLLM’s training data, so the score reflects generalization to unseen business logic rather than memorization.
How does PhoneLLM compare to GPT 5.6 Terra?
On the PhoneBench v1 leaderboard, PhoneLLM scores comparably to GPT 5.6 Terra, one of the models commonly used in production voice agent deployments today, while being about 94% cheaper to run and delivering a P95 time-to-first-token roughly 1,300ms faster. That latency gap matters more in voice than in most other LLM applications, because it’s the difference between an agent that feels responsive and one that produces awkward silence on every turn.
The cost comparison is where the mixture-of-experts architecture pays off directly. Because only 3.5B of PhoneLLM’s 30B parameters are active per token, it can serve far more concurrent conversations per GPU than a dense model of similar total size, and far more cheaply than most large hosted frontier models. Pipecat’s benchmarking sweeps found that Nemotron 3 Nano-class architecture (the same one underlying PhoneLLM) can sustain roughly 44 concurrent inference processes per NVIDIA B200 while staying under a 600ms P95 time-to-first-answer-token target, translating to about 88 concurrent voice agent sessions per B200 node. Using Modal’s published B200 pricing with region pinning and a 70% utilization target, Pipecat calculates an effective cost of about $0.00025 per agent-minute, well below typical hosted API pricing for comparable accuracy.
Not every model benefits from this kind of efficiency. Pipecat’s cost data flags Gemini 3.6 Flash as a notable outlier: even with “minimal” thinking settings, it still generates a large volume of thinking tokens, which drives its effective cost per minute up regardless of its sticker price per token.
Is self-hosting a smaller model actually worth it?
The core tradeoff Pipecat lays out is what they call “model size arbitrage.” Large hosted frontier models benefit from enormous economies of scale, and for general-purpose tasks, using an API is often still cheaper than self-hosting. But for a narrow, well-defined workload like phone agent conversations, a smaller model tuned tightly to that task can match or beat a frontier model’s accuracy while using a fraction of the compute per response. The savings come specifically from picking a model sized to the job rather than defaulting to the biggest available option.
Self-hosting also opens up latency optimizations that aren’t available with a closed API. Because PhoneLLM runs on open inference servers like vLLM and SGLang, teams can tune concurrency, caching, and hardware placement for their specific latency target rather than accepting whatever tradeoff a hosted provider has chosen. Pipecat reports single-request P95 time-to-first-token under 100ms on a single B200 in their own testing, and notes that Modal’s Auto Endpoints configuration, tuned specifically for PhoneLLM’s conversational workload, roughly doubles maximum concurrency compared to a generic vLLM cookbook setup at the same sub-600ms latency target.
The catch is operational overhead: self-hosting means owning GPU capacity planning, inference server configuration, and monitoring, which is nontrivial work that a hosted API abstracts away. For teams already running voice infrastructure at meaningful volume, that tradeoff increasingly favors self-hosting a fit-for-purpose model like PhoneLLM.
How do you run PhoneLLM?
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
The model weights are published on Hugging Face under pipecat-ai/phonellm-alpha-1, released under a BSD 2-Clause license (as a derivative of NVIDIA’s Nemotron Open Model License) with no commercial usage restrictions. It can be served with either vLLM or SGLang using standard Nemotron 3 Nano recipes, with trust_remote_code=True required. Pipecat stresses two settings as non-negotiable for matching the model’s trained behavior: temperature should be set to zero, and thinking must be disabled via the chat template. For teams that don’t want to manage their own GPU cluster, Modal offers PhoneLLM as a one-command Auto Endpoint deployment, with a configuration co-developed with Pipecat specifically for low-latency conversational workloads.
Frequently Asked Questions
What base model is PhoneLLM Alpha 1 built on?
It’s a full-parameter fine-tune of NVIDIA’s Nemotron 3 Nano 30B-A3B, a hybrid Mamba-Transformer mixture-of-experts model with 30 billion total parameters and 3.5 billion active parameters per token, trained further using the NVIDIA NeMo framework.
Is PhoneLLM Alpha 1 free to use commercially?
Yes. It’s released under a BSD 2-Clause license as a derivative of NVIDIA’s Nemotron Open Model License, and Pipecat states there are no commercial restrictions on its use.
How much cheaper is PhoneLLM than a frontier model like GPT 5.6 Terra?
On Pipecat’s PhoneBench v1 benchmark, PhoneLLM matches GPT 5.6 Terra’s accuracy at roughly 94% lower cost, with a P95 time-to-first-token about 1,300ms faster.
Why does PhoneLLM disable thinking/reasoning tokens?
Reasoning tokens add latency between a caller’s utterance and the agent’s response, which breaks the tight timing voice conversations require. PhoneLLM was trained and evaluated specifically with thinking disabled, so it’s tuned to call tools accurately and respond correctly without that extra reasoning step.
What is PhoneBench and how is it different from typical LLM benchmarks?
PhoneBench v1 is Pipecat’s benchmark for phone voice agents. It uses a panel of LLM judges, calibrated against human labels, to score subjective qualities like speaking style, tool call accuracy, say/do consistency, and caller outcome, alongside measured latency and estimated cost per minute, on scenarios kept separate from PhoneLLM’s training data.
