TreasureX Researchcs.AIAudio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model
Harsha Vardhan Khurdula, Abhinav Kumar Singh, Yoeven D Khemlani, Vineet Agarwal · July 14, 2026
A new approach to automatic speech recognition (ASR) uses a discrete diffusion language model to transcribe speech in parallel, rather than the usual token-by-token autoregressive method. The system, built on DiffusionGemma (a 26B mixture-of-experts model), generates entire transcripts by gradually denoising random tokens. A frozen Whisper encoder extracts audio features, a lightweight projector maps them into the model's embedding space, and low-rank adapters allow the frozen backbone to handle audio. Only 0.16% of the backbone parameters (42M) are trained. The key challenge was that standard training objectives failed to ground audio because gradients couldn't reach the projector through the frozen attention layers. A connectionist temporal classification (CTC) loss applied through the frozen output head solved this. The model achieves 6.6% word error rate on LibriSpeech test-clean, transcribes in about 8 parallel steps regardless of length, and uses a single adapter trained on six languages (tested on English, Hindi, Mandarin). This work shows that discrete diffusion models can be effective for ASR, potentially reducing latency and simplifying multilingual systems.
Read more: https://arxiv.org/abs/2607.13013
Educational content only, generated with AI assistance from public research abstracts. Not investment advice. Thank you to arXiv for use of its open access interoperability.