How to Deploy PhoneLLM Alpha 1 with vLLM, SGLang, or Modal
A practical guide to self-hosting PhoneLLM Alpha 1 for voice agents, covering vLLM and SGLang settings, hardware needs, and Modal AutoEndpoints.
What is PhoneLLM Alpha 1 and why self-host it?
PhoneLLM Alpha 1 is an open-weight language model built by the Pipecat team at Daily specifically for voice agent workloads like phone-based customer service and outbound calling. It’s a full-parameter fine-tune of NVIDIA’s Nemotron 3 Nano 30B-A3B, a hybrid Mamba-Transformer mixture-of-experts model with 30 billion total parameters but only 3.5 billion active per token. Because the weights are public under a BSD 2-Clause license with no commercial restrictions, you can run it on your own GPUs, on Modal, or on any infrastructure that supports a 30B open-weight model.
TL;DR
- PhoneLLM Alpha 1 is a Nemotron 3 Nano 30B-A3B fine-tune trained with NVIDIA NeMo specifically for accurate tool calling and low-latency multi-turn phone agent conversations.
- The model fits comfortably on a single B200 GPU, with single-request P95 time-to-first-token under 100ms and room left over for long context and KV cache.
- Correct inference requires two non-negotiable settings: temperature=0 and thinking disabled via
chat_template_kwargs: {"enable_thinking": false}, since the model was trained under those exact conditions. - You can serve it with standard Nemotron 3 Nano recipes in vLLM or SGLang, using
trust_remote_code=True, or deploy it in a few clicks with Modal AutoEndpoints. - Modal’s tuned configuration for PhoneLLM roughly doubles max concurrency compared to the generic vLLM cookbook setup, while holding P95 time-to-first-answer-token under 600ms.
- Cost modeling from Pipecat’s benchmark spreadsheet estimates a per-minute agent cost near $0.00025 on a B200 at 70% utilization with 88 concurrent agent sessions per node.
- The model supports a 262,144-token context window in bfloat16 precision, which matters for long multi-turn calls with tool history and system prompts.
What hardware do you need to run PhoneLLM Alpha 1?
PhoneLLM was designed around NVIDIA’s Nemotron 3 architecture, which the Pipecat team chose specifically because it scales efficiently on modern NVIDIA hardware. According to the model card, PhoneLLM “fits nicely and runs very fast on a single NVIDIA B200,” with headroom left for long contexts and context caching even at that scale. Single-request P95 time-to-first-token comes in under 100ms on that hardware.
For production deployments serving many concurrent calls, the calculus changes. Pipecat’s benchmarking found that Nemotron 3 Nano based models (PhoneLLM included) can support up to 44 concurrent inference processes per B200 while staying under a 600ms P95 time-to-first-answer-token target, which works out to 88 concurrent agent sessions per node when processes are paired appropriately. That efficiency is a direct result of the model’s mixture-of-experts design: only 3.5B of the 30B parameters activate per token, which keeps per-request compute low even at high concurrency.
If you’re experimenting rather than running production traffic, a single B200 (or comparable high-memory GPU capable of serving a 30B MoE model in bfloat16) is enough to get started.
How do you deploy PhoneLLM with vLLM or SGLang?
The Pipecat team runs PhoneLLM in production using both SGLang and vLLM, and recommends starting from the standard Nemotron 3 Nano serving recipes for each:
- SGLang: use the documented Nemotron 3 Nano configuration from SGLang’s cookbook as your baseline.
- vLLM: use the Nemotron 3 Nano recipe published in the vLLM recipes repository.
Because PhoneLLM uses a custom architecture inherited from Nemotron 3 Nano, you need to set trust_remote_code=True when loading the model in either framework. The weights are published in bfloat16 safetensors format on Hugging Face under pipecat-ai/phonellm-alpha-1.
The two settings that matter most, regardless of which server you use, are:
- temperature=0
- Thinking disabled, passed via
chat_template_kwargs: {"enable_thinking": false}
These aren’t just suggestions. PhoneLLM was trained under these exact conditions, and the model card is explicit that these settings “align with how the model was trained.” Running with thinking enabled or with a non-zero temperature will produce results that don’t match the model’s tuned behavior, particularly around tool-call accuracy, which is the core capability PhoneLLM was built to nail.
Beyond those two settings, the generic Nemotron 3 Nano configurations are a reasonable starting point for context length, batching, and memory settings, since PhoneLLM shares the same underlying architecture.
How do you deploy PhoneLLM on Modal?
For teams that don’t want to manage GPU infrastructure directly, PhoneLLM Alpha 1 is available through Modal’s AutoEndpoints system. You can spin up a deployment either through the Modal Endpoints dashboard with a few clicks, or from the command line:
modal endpoint create --model pipecat-ai/phonellm-alpha-1Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
What makes the Modal deployment notable isn’t just convenience. Modal worked directly with the Pipecat team to tune the AutoEndpoints configuration using representative voice agent workload data, rather than relying on a generic serving setup. The result, per the model card, is a configuration that “approximately doubles the max agent concurrency” compared to the standard vLLM cookbook configuration, while still holding P95 time-to-first-answer-token under 600ms. For a workload where latency budgets are extremely tight, that kind of workload-specific tuning has a real effect on how many concurrent calls a single GPU can economically support.
Why does inference latency matter this much for voice agents?
Voice agents have a much tighter latency budget than most LLM applications. Pipecat’s own data indicates that voice-to-voice latency, the total round-trip from a caller finishing speech to hearing the agent’s response, needs to stay around 1,500ms at the P95 level for people to be comfortable with the interaction. That number has to account for network overhead, audio processing, speech-to-text, application logic, the LLM call itself, and text-to-speech.
In a well-optimized voice agent, Pipecat’s latency breakdown allocates roughly 650ms of that budget to LLM time-to-first-token specifically. That’s a tight window. For context, the model card notes that GPT 5.6 Terra’s P95 time-to-first-token running in fast mode is around 1,900ms, which alone would blow past the entire 1,500ms voice-to-voice target before accounting for STT, TTS, or network latency.
This is the core argument for using a purpose-built, smaller open-weight model instead of a general-purpose frontier model for phone agent use cases: latency and cost scale with model size and serving configuration, and a model tuned for fast, non-reasoning responses can hit targets that reasoning-heavy frontier models structurally cannot without extra engineering.
How much does self-hosting PhoneLLM cost compared to APIs?
Cost modeling for conversational agents is inherently about cost per minute, not just cost per token, since sessions run long and involve many turns of varying length. Pipecat published a cost estimator spreadsheet modeling per-minute costs across multiple LLMs based on observed inference patterns from their PhoneBench benchmark.
Using Modal’s B200 pricing as the reference point: a B200 costs $6.2496 per hour as a base rate. Adding a 1.5x multiplier for region pinning brings that to $9.3744/hour. Targeting 70% utilization brings the effective cost to $13.392/hour, or $0.2232/minute. Divided across 88 concurrent agent sessions per B200 node (the max concurrency that keeps P95 time-to-first-answer-token under the 600ms target), that works out to a per-minute agent cost of roughly $0.00025.
The broader pattern in Pipecat’s cost data is that bigger, more general-purpose models cost more to run, and the savings from self-hosting mostly come from choosing a smaller model that’s actually well-matched to the task rather than from self-hosting itself. Large API providers operate at scale and are efficient; the win from self-hosting PhoneLLM comes specifically from using a 3.5B-active-parameter model tuned for phone agent tasks instead of a much larger general-purpose model.
Frequently Asked Questions
What base model is PhoneLLM Alpha 1 built on?
It’s a full-parameter fine-tune of NVIDIA’s Nemotron 3 Nano 30B-A3B, a hybrid Mamba-Transformer mixture-of-experts model, trained using the NVIDIA NeMo framework.
What inference settings does PhoneLLM require?
Set temperature to 0 and disable thinking mode via chat_template_kwargs: {"enable_thinking": false}. These match the exact conditions the model was trained under, and deviating from them affects output quality, particularly tool-call accuracy.
Can I run PhoneLLM Alpha 1 without a B200 GPU?
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
The model card specifically documents performance on a single B200, where it runs fast with room for long context and caching. Other GPUs capable of serving a 30B-parameter bfloat16 MoE model may work, but published latency and concurrency figures are based on B200 hardware.
Does PhoneLLM support long conversations?
Yes. It has a 262,144-token context length, which supports long multi-turn calls, extended tool-call histories, and detailed system prompts common in phone agent deployments.
Is PhoneLLM free to use commercially?
It’s released under a BSD 2-Clause license with no commercial restrictions, though it’s a derivative work covered in part by NVIDIA’s Nemotron Open Model License, so it’s worth reviewing both license files before deployment.
