mlx-audio STT — Listening on Apple Silicon

Listening on Apple Silicon

The flip side of TTS — STT (speech-to-text) — also lives in mlx-audio. The dominant family is Whisper and its derivatives; mlx-audio ships MLX-optimized loaders for Whisper-class models that take an audio file in and return transcribed text out.

The quality story for STT is more mature than TTS. Whisper-large-v3 (and its Apple-Silicon-friendly distillations) is genuinely strong on most languages, and the transcription latency on M-Pro and up is well below realtime — you can transcribe an hour of audio in minutes.

Install + minimum loop

Same install (mlx-audio package). The Python loop loads a Whisper-class model, points it at a WAV/MP3/M4A file, and gets back text plus per-segment timestamps if you want them.

What you get back

STT typically returns more than just text. The structured response usually includes segments (chunks of audio with start / end times), language detection, and per-token confidence scores if the model exposes them. Use the segments when you need to align text back to audio (subtitles, search-by-time); use just the joined text when you only want a transcript.

Realtime factor

The single most useful number for STT performance is the realtime factor — how many seconds of audio you can transcribe per second of wall-clock time. On an M-Ultra with a Whisper-large MLX model, expect a realtime factor in the 5–15× range for typical audio. On an M-Pro, more like 2–5×. Realtime factor < 1 means you can't keep up with a live stream; realtime factor > 1 means you can transcribe live audio plus have headroom.

The diarization gap

STT (Whisper-class) gives you what was said, not who said it. Speaker diarization — labeling each segment with a speaker id — is a separate capability and isn't yet first-class in mlx-audio as of 2026-05. If you need diarization, the typical workflow is to run STT in mlx-audio for the transcripts and pyannote-audio (or similar, on PyTorch) for the diarization, then merge the two streams. Worth knowing about; not a single-tool problem on MLX yet.

Code

Transcribe an audio file with a Whisper-class MLX model·python

# Pattern (exact API surface depends on mlx-audio version):
from mlx_audio.stt.generate import generate as transcribe

result = transcribe(
    audio_path="path/to/audio.wav",
    model_path="mlx-community/whisper-large-v3-mlx",   # or current best-of-class STT model
    output_path="transcript.txt",
)

# `result` is typically a dict-like object with:
#   .text       — full transcribed text
#   .segments   — list of {start, end, text} per segment
#   .language   — auto-detected language code (e.g. 'en', 'ko')
print(result.text[:500])

Measure the realtime factor on your machine·python

import time, wave
from mlx_audio.stt.generate import generate as transcribe

audio_path = "path/to/audio.wav"

# Total audio duration via wave module (works for WAV; for MP3/M4A use ffprobe)
with wave.open(audio_path, "rb") as wf:
    duration_sec = wf.getnframes() / float(wf.getframerate())

t0 = time.perf_counter()
result = transcribe(audio_path=audio_path, model_path="mlx-community/whisper-large-v3-mlx")
elapsed = time.perf_counter() - t0

print(f"audio duration : {duration_sec:.1f} s")
print(f"transcribe wall : {elapsed:.1f} s")
print(f"realtime factor : {duration_sec / elapsed:.1f}x")

# Sample expected (M3 Ultra Studio with whisper-large-v3-mlx):
#   realtime factor : ~10x   (varies with audio difficulty and model)

Exercise

Pick a representative audio sample from your actual workflow (a podcast you listen to, a meeting recording, anything 5–15 minutes). Run STT on it with the realtime-factor measurement. Compare to your gut — is the result better or worse than you expected? If you have a multilingual or noisy sample, run that too and note how the realtime factor and quality both change. Two sentences on what you'd plan capacity for.