How to Run Audio8 ASR Infinite Locally with vLLM or Docker
Step-by-step guide to self-hosting Audio8 ASR Infinite for 24/7 streaming transcription using Docker, vLLM, or Torch inference.
What is Audio8 ASR Infinite?
Audio8 ASR Infinite is an open-weight streaming speech recognition model released by Edge0 under the Apache 2.0 license. It’s built for continuous, low-latency transcription rather than batch processing of pre-recorded files. The model decodes audio 12.5 times per second at its fastest setting and uses a rolling KV cache so it can run 24/7 without memory growth or accuracy drift, even though its native context window is only 30 seconds. It supports Chinese and English, and ships with a semantic voice activity detection (VAD) system that can tell the difference between a thinking pause, a stutter, and an actual end of turn, something traditional acoustic VAD often gets wrong.
The model is available on Hugging Face (Edge0/Audio8-ASR-Infinite) and on GitHub (Edge0-AI/Audio8-ASR-Infinite), and it can be deployed either through a Docker Compose stack backed by an adapted vLLM build, or through direct Torch inference for simulated streaming decode.
TL;DR
- Audio8 ASR Infinite is a streaming ASR model that transcribes continuously without the latency or memory growth typical of long-context transformers.
- It uses a rolling KV cache with exact RoPE re-basing over a 30-second window, which is what lets it run indefinitely instead of resetting or slowing down.
- Three audio clock options (80ms, 120ms, 160ms) let you trade responsiveness for compute cost, each paired with specific delay settings that were actually post-trained.
- The recommended deployment path is Docker Compose, which stands up both the vLLM-based realtime server and a web demo client in one command.
- On Chinese benchmarks like aishell1 and aishell4, the model posts substantially lower character error rates than comparable streaming models such as Voxtral-Mini-4B-Realtime and Nemotron-3.5-ASR-Streaming, though it trails slightly on English LibriSpeech word error rate.
- A semantic VAD head with multiple prediction horizons (0.5 to 3.0 seconds) is bundled in the checkpoint, separate from the core transcription weights.
- The architecture borrows the Voxtral realtime audio tower and pairs it with a Qwen2.5-3B-Instruct decoder, giving it a total footprint of about 8.17 GB in bfloat16.
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
How does the streaming architecture actually work?
Audio8 ASR Infinite inherits its audio tower from Voxtral Realtime 4B and its decoder from Qwen2.5-3B-Instruct, connected through a trained audio projector and a frame length embedding module. The audio tower has 32 layers with a hidden size of 1280 and works on 128 mel bins with a sliding window of 750. The text decoder has 36 layers, hidden size 2048, using 16 query heads and 2 KV heads in a grouped-query attention setup. The projector caps frame length at 8, producing a projection size of 10240 with GELU activation.
The key design choice is the “audio clock.” Instead of processing a fixed chunk and returning one transcript, the model emits one text token per clock step. At 80ms clock, that’s 12.5 decisions per second; drop to 120ms and it’s about 8.3; at 160ms it’s 6.25. Each option pairs with specific frame_len and streaming_n_left_pad_tokens settings that were the ones the model was actually post-trained on:
| audio clock | frame_len | left pad tokens | supported delays |
|---|---|---|---|
| 80 ms | 4 | 18 | 240 / 320 / 480 / 560 ms |
| 120 ms | 6 | 12 | 240 / 480 ms |
| 160 ms | 8 | 9 | 320 / 480 ms |
You can technically mix other combinations, but the model card is explicit that only these pairings were optimized. The target_delay_ms value has to be an integer multiple of whichever clock you pick, which is what lets you dial in more accuracy at the cost of responsiveness, or vice versa.
Why does the rolling KV cache matter for 24/7 use?
Most transformer-based ASR models have a fixed context window and either truncate, reset, or slow down as audio gets longer. Audio8 ASR Infinite’s checkpoint has a native context of just 30 seconds, but the rolling KV cache with exact RoPE re-basing keeps that window sliding forward indefinitely. Memory usage and per-token latency stay flat whether you’ve been streaming for 30 seconds or 30 hours. That’s the specific engineering claim behind the “Infinite” in the name, and it’s also why the vLLM deployment path exists separately from the plain Torch example: sustained, always-on transcription is a server workload, not a one-shot script.
How do you set it up with Docker and vLLM?
The documented, canonical path is Docker Compose. From the docker directory in the GitHub repo, you point an environment variable at your local checkpoint and bring the stack up:
cd docker
AUDIO8_MODEL_DIR=/path/to/checkpoint docker compose up -dThis single command starts the adapted vLLM server and a web demo client together. Once it’s running, you can verify the deployment through a browser at http://localhost:8080/ for plain HTTP or https://localhost:8443/ for a TLS-proxied connection (you’ll need to accept a self-signed certificate on first load).
If you’d rather drive it from the terminal instead of a browser, the repo includes a WebSocket client example:
python -m audio8_asr_infinite.examples.vllm_realtime_client \
--ws-url ws://127.0.0.1:18191/v1/realtime \
--audio sample.wav --language zh --target-delay-ms 480 --paceBuilt like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
Note the port distinction: 18191 is the host-published port defined in docker-compose.yml, while the service itself listens internally on 18190 inside the Docker network. This is the setting most likely to trip people up if they’re editing the compose file or trying to reach the service from another container.
How do you run it without Docker, using plain Torch?
For simpler testing, or if you don’t want to manage a container stack, there’s a direct Torch-based streaming decode script:
python -m audio8_asr_infinite.examples.torch_streaming_decode \
--checkpoint /path/to/checkpoint \
--audio sample.wav --language zh --transcription-delay-ms 480This simulates streaming decode on a single audio file rather than serving a persistent socket, which makes it a reasonable way to sanity-check a checkpoint or benchmark decode quality before committing to the full server deployment. The model card also includes a lower-level Python example using simulated_streaming_greedy_decode_batch directly, which is useful if you need to integrate the model into a custom inference pipeline rather than using either of the packaged examples. One constraint to flag: the model only supports a full merged weight directory as distributed. Adapter-style weights or partially converted checkpoints won’t work.
Is Audio8 ASR Infinite worth it compared to alternatives?
On the numbers Edge0 published, it’s a strong performer on Chinese transcription specifically. At a 480ms delay and 80ms frame length, it posts a character error rate of 1.750 on aishell1/test and 2.893 on aishell4/test, well ahead of Voxtral-Mini-4B-Realtime-2602 (16.795 and 16.456 on the same sets) and Nemotron-3.5-ASR-Streaming-0.6b (12.927 and 14.677, measured at 560ms delay). On English, the picture flips: Voxtral edges it out on both LibriSpeech test-clean (2.210 vs 3.042 word error rate) and test-other (5.552 vs 6.808). Averaged across all four sets, Audio8 ASR Infinite comes out ahead at 3.623 versus 10.253 for Voxtral and 9.524 for Nemotron, but that average is doing some work given the uneven per-language performance.
Whether it’s worth deploying depends on your use case. If you need bilingual streaming with strong Chinese accuracy and a genuinely unbounded runtime, it’s a solid fit. If your workload is English-only, a model tuned specifically for that language may edge it out on raw word error rate.
Frequently Asked Questions
What hardware do I need to run Audio8 ASR Infinite?
The model card doesn’t publish an explicit VRAM requirement, but the checkpoint is roughly 8.17 GB in bfloat16 for the main weights plus a separate semantic VAD head file. Running it via the vLLM Docker stack implies a CUDA-capable GPU is expected for real-time performance, consistent with the .cuda() call in the example code.
Does Audio8 ASR Infinite support languages other than Chinese and English?
No. The model card states it is bilingual, supporting only Chinese and English at this stage.
What’s the difference between the vLLM path and the Torch path?
The vLLM Docker Compose stack is meant for sustained, real-time, 24/7 serving with a WebSocket interface and a bundled web demo. The Torch script (torch_streaming_decode) runs simulated streaming decode on a single audio file, which is better suited to testing and evaluation than production serving.
Is the semantic VAD feature fully available in this release?
Partially. This is labeled a preview release delivering the core streaming transcription. The semantic VAD heads (8 classes, with prediction horizons from 0.5 to 3.0 seconds) are included in the checkpoint, but frame-level semantic perception beyond transcription is listed as in progress toward a formal release.
Can I use adapter weights or a partial checkpoint instead of the full model?
No. The documentation is explicit that only a full merged weight directory is supported. Adapter-style or partially converted weights will not work with the provided inference code.