Audio8 ASR Infinite: Open Streaming Speech Recognition That Never Stops
Audio8 ASR Infinite is an open-weight streaming ASR model built for 24/7 transcription. Here's how its rolling KV cache works and how it benchmarks.
What is Audio8 ASR Infinite?
Audio8 ASR Infinite is an open-weight streaming speech recognition model released by Edge0 under Apache 2.0. It transcribes Chinese and English audio in real time and, thanks to a rolling KV cache, can keep transcribing indefinitely without the memory or latency growth that normally limits streaming transformers. It ships as a Hugging Face model plus a Docker/vLLM deployment path, so it can run on local hardware as a continuous transcription service rather than a one-shot batch job.
TL;DR
- Audio8 ASR Infinite decodes speech 12.5 times per second at its fastest setting and offers a selectable audio clock of 80, 120, or 160 ms depending on how much responsiveness versus compute cost you need.
- Its rolling KV cache keeps a fixed 30-second context window with exact RoPE re-basing, which holds memory and latency constant even when the model runs for hours or days straight.
- On Chinese test sets (aishell1, aishell4) it posts noticeably lower character error rates than Voxtral-Mini-4B-Realtime and Nemotron 3.5 ASR streaming, while trailing slightly on English LibriSpeech word error rates.
- The architecture reuses a Voxtral Realtime 4B audio tower paired with a Qwen2.5-3B-Instruct decoder, with the projector and frame-length embeddings trained from scratch.
- It includes a semantic VAD head that tries to tell apart thinking pauses, stutters, and actual end-of-turn, which is a common failure point for traditional acoustic voice activity detection.
- Deployment is built around Docker Compose and a patched vLLM build, exposing a WebSocket realtime endpoint plus a browser demo client.
- The current release is explicitly a preview: it covers streaming transcription only, with frame-level semantic perception features planned for a formal release.
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
How does the rolling KV cache enable 24/7 transcription?
Most streaming transformer ASR models hit a wall because their key-value cache grows with every second of audio processed. Eventually memory fills up, or the model has to truncate context in ways that hurt accuracy or introduce latency spikes. Audio8 ASR Infinite sidesteps this with a rolling KV cache tied to a native 30-second context window. Instead of letting the cache grow unbounded, it continuously evicts old frames and re-bases the rotary position embeddings (RoPE) so the model’s internal sense of “where am I in time” stays mathematically consistent even though the underlying audio window keeps sliding forward.
The practical result: memory footprint and per-token latency stay flat whether the model has been running for one minute or one day. That’s the core claim behind the “Infinite” in the name, and it’s what makes the model suitable for always-on use cases like call center monitoring, live captioning for broadcast, or continuous meeting transcription, where a model that periodically stalls or leaks memory isn’t viable.
What’s under the hood architecturally?
Audio8 ASR Infinite is built by combining pieces from two existing model families rather than training a network from scratch. The causal audio tower is initialized from Voxtral Realtime 4B (32 layers, hidden size 1280, 128 mel bins, sliding window of 750) and fine-tuned further. The text decoder is initialized from Qwen2.5-3B-Instruct (36 layers, hidden size 2048, 16 query heads and 2 KV heads) and also trained further, along with its LM head. Two components are trained entirely from random initialization: the audio projector, which maps audio tower output into the decoder’s embedding space (max frame length 8, projection size 10240, GELU activation), and a frame-length embedding module that lets the model condition its behavior on which audio clock setting is active.
This frame-length conditioning is what allows a single checkpoint to support multiple streaming granularities (80/120/160 ms) rather than requiring separate models for each. There’s also a dedicated semantic VAD head, stored as a separate safetensors file, with 8 output classes and prediction horizons at 0.5, 1.0, 2.0, and 3.0 seconds, aimed at distinguishing genuine end-of-turn from mid-sentence pauses or disfluencies.
Total weights come to about 8.17 GB in bfloat16, plus the smaller semantic VAD head file, and the vocabulary size (151,936 tokens) matches Qwen2.5’s tokenizer, which is expected given the decoder lineage.
How accurate is it compared to other streaming ASR models?
The model card reports benchmarks at an 80 ms audio clock with a 480 ms transcription delay (6 delay tokens), using greedy decoding with end-of-sequence suppressed. Compared against Voxtral-Mini-4B-Realtime-2602 and Nemotron 3.5 ASR streaming (0.6B), the results are mixed by language:
- Chinese (character error rate): Audio8 ASR Infinite scores 1.750 on aishell1/test and 2.893 on aishell4/test, both well below Voxtral’s 16.795 and 16.456 on the same sets, and below Nemotron’s 12.927 and 14.677 (measured at 560 ms delay).
- English (word error rate): On LibriSpeech test.clean, Audio8 ASR Infinite scores 3.042 versus Voxtral’s 2.210, a case where Voxtral comes out ahead. On test.other, Audio8 ASR Infinite scores 6.808 versus Voxtral’s 5.552, another Voxtral win.
- Overall average: Audio8 ASR Infinite’s average across all four sets is 3.623, compared to Voxtral’s 10.253 (calculated over its two reported sets) and Nemotron’s 9.524.
The pattern suggests Audio8 ASR Infinite was tuned heavily for Chinese transcription accuracy, where it posts a decisive lead, while giving up a small amount of English accuracy relative to Voxtral. For bilingual deployments where Chinese is a priority, that trade favors Audio8 ASR Infinite; for English-only pipelines, Voxtral’s numbers are closer or better on raw WER.
How do you deploy it locally with vLLM?
The documented path is Docker Compose, which spins up a patched vLLM serving stack and a bundled web demo in one command:
cd docker
AUDIO8_MODEL_DIR=/path/to/checkpoint docker compose up -dOnce running, the web client is reachable at http://localhost:8080/ (plain HTTP) or https://localhost:8443/ (TLS, self-signed certificate). The realtime service itself listens on port 18190 inside the compose network but is published on host port 18191, exposing a WebSocket endpoint at /v1/realtime. A terminal client is included for testing without the browser:
python -m audio8_asr_infinite.examples.vllm_realtime_client \
--ws-url ws://127.0.0.1:18191/v1/realtime \
--audio sample.wav --language zh --target-delay-ms 480 --paceFor experimentation outside the vLLM stack, the model also supports a simulated streaming decode path directly through Transformers, using the Audio8ASRInfiniteForConditionalGeneration class with a custom audio config object specifying samples-per-token, padding tokens, and sampling rate. This is slower and not meant for production, but it’s useful for testing accuracy on saved audio files without standing up the full Docker service. Only fully merged weight checkpoints are supported; partial or adapter-based weight conversions aren’t.
Is Audio8 ASR Infinite worth using right now?
For teams that need continuous, self-hosted transcription and can work in Chinese or English, it’s a strong candidate, particularly if Chinese accuracy matters, given the large CER gap over Voxtral and Nemotron on aishell benchmarks. The Apache 2.0 license and full Docker/vLLM deployment path make it straightforward to self-host without depending on a cloud ASR API, which matters for cost control and data privacy in always-on use cases like meeting transcription, live captioning, or call monitoring.
The caveats: this is a preview release. The semantic VAD head is present in the weights but the roadmap describes fuller “frame-level semantic perception” as still in progress for a formal release. English WER lags Voxtral slightly, so English-heavy pipelines may want to benchmark both. And running vLLM with a patched build via Docker requires GPU infrastructure and some comfort with containerized ML serving, so it’s not a drop-in API replacement for teams without local GPU capacity.
Frequently Asked Questions
What languages does Audio8 ASR Infinite support?
It’s bilingual, supporting Chinese and English transcription, with benchmarks reported separately for aishell (Chinese) and LibriSpeech (English) test sets.
What does “unlimited-length transcription” actually mean here?
It means the model can run continuously on live audio without the memory usage or latency growing over time. This is achieved via a rolling KV cache with a fixed 30-second window and exact RoPE re-basing, rather than by extending the model’s native context length.
How is latency controlled during streaming?
Through two settings: the audio clock (80, 120, or 160 ms, controlling how often the model makes a decoding decision) and the transcription delay (240 to 560 ms, controlling how many tokens of lookahead the model uses before committing to text). Longer delays generally trade responsiveness for accuracy.
How does it compare to Voxtral and Nemotron on accuracy?
Remy doesn't write the code. It manages the agents who do.
Remy runs the project. The specialists do the work. You work with the PM, not the implementers.
It substantially outperforms both on Chinese character error rate across aishell1 and aishell4 test sets. On English LibriSpeech word error rate, Voxtral-Mini-4B-Realtime scores slightly better on both test.clean and test.other splits.
Can it be self-hosted without cloud infrastructure?
Yes. The documented deployment path uses Docker Compose to run a patched vLLM server plus a web demo client locally, exposing a WebSocket realtime API on a local port, which keeps all audio processing on your own hardware.