Microsoft VibeVoice Speech-to-Text Model Tested

Microsoft has introduced VibeVoice, an audio model similar to Whisper that converts speech to text. The tool includes speaker diarization directly in the model and carries an MIT license. The company made it available on January 21st, 2026.

Simon Willison, a well-known developer and creator of the Datasette data browser, tested the model recently. He ran it on his 128GB M5 Max MacBook Pro. Willison processed a downloaded copy of his appearance on Lenny Rachitsky's podcast, a popular show focused on product management and technology.

Command for Quick Transcription

Willison used a single command line with uv, mlx-audio developed by Prince Canuma, and the 5.71GB mlx-community/VibeVoice-ASR-4bit version. This quantized edition comes from the original 17.3GB VibeVoice-ASR model. The exact command follows:

uv run , with mlx-audio python -m mlx_audio.stt.generate \
, model mlx-community/VibeVoice-ASR-4bit \
, audio lenny.mp3 , output-path lenny \
, format json , verbose , max-tokens 32768

Results showed processing in 524.79 seconds. The prompt phase used 26615 tokens at 50.718 tokens per second. Generation handled 20248 tokens at 38.585 tokens per second. Peak memory reached 30.44 GB.

That equaled 8 minutes and 45 seconds for one hour of audio. Willison confirmed it worked on both .wav and .mp3 files.

Microsoft's push into AI models builds on its collaboration with OpenAI, which created the original Whisper model in 2022. Whisper set benchmarks for automatic speech recognition, handling multiple languages and accents. VibeVoice follows that approach with added speaker separation, useful for meetings, podcasts, and interviews.

Memory and Token Details

Without specifying , max-tokens, the model defaults to 8192 tokens, covering about 25 minutes of speech. Willison learned this through experiments and increased it fourfold to capture the full hour.

The command reported 30.44 GB peak RAM. Activity Monitor displayed higher usage: 61.5 GB during prefill and around 18 GB during generation.

Simon Willison maintains a blog tracking AI and database tools. He co-authored Django, a Python web framework, and developed Datasette for exploring structured data through SQL queries in browsers.

JSON Output Structure

The transcription produced JSON as an array of objects. Each entry includes text, start time, end time, duration, and speaker_id. Examples include:

{
"text": "And an open question for me is how many other knowledge work fields are actually prone to these agent loops?",
"start": 13.85,
"end": 19.5,
"duration": 5.65,
"speaker_id": 0
}

{
"text": "Now that we have this power, people almost underestimate what they can do with it.",
"start": 19.5,
"end": 22.78,
"duration": 3.280000000000001,
"speaker_id": 1
}

{
"text": "Today, probably 95% of the code that I produce, I didn't type it myself. I write so much of my code on my phone. It's wild.",
"start": 22.78,
"end": 30.0,
"duration": 7.219999999999999,
"speaker_id": 0
}

Willison loaded the JSON into Datasette Lite for easy viewing.

Speaker Detection and Limits

Datasette Lite revealed three speakers. It distinguished Willison and Rachitsky in the main talk, plus a separate voice for Rachitsky's intro and sponsor segments.

VibeVoice processes up to one hour of audio per run. The test covered just the first hour of the podcast. For longer files, users must split the audio with about a minute of overlap to prevent cut-off words. Then, align speaker IDs across segments.

This account comes from Simon Willison's post dated April 27, 2026.

Related on Neura Market

• AI Models Directory
• Speech-to-Text Tools
• Open-Source AI Projects