Realtime Speech-to-Text on macOS — Three Architectures Compared

When we set out to add realtime subtitles to Yfli Screen Recorder, we threw out our architecture three times. This article walks through all three approaches — pure cloud streaming, on-device Whisper, and hybrid edge-cloud — covering the data flow, latency budget, and engineering pitfalls of each. If you’re adding live captions to your own app, this will save you months.

Realtime Speech-to-Text Is More Than "Speech to Text"

Most people think ASR (Automatic Speech Recognition) is just “feed audio, get text.” But once you need realtime, speaker-labeled, multilingual output, it becomes a full pipeline. Any weak link breaks the experience.

Microphone (16kHz / mono PCM, 20ms frames)
   │
   ▼
Noise Suppression / Voice Isolation
   │  Remove keyboard clicks, fan noise, background music
   ▼
VAD (Voice Activity Detection)
   │  Is someone speaking right now? Segment into utterances.
   ▼
Streaming ASR
   │  Output partial (draft) + final (committed) text as audio arrives
   ▼
Speaker Diarization
   │  Label "Speaker A said this, Speaker B said that"
   ▼
Subtitle Rendering / State Machine
      partial = gray → final = black → speaker label appended

Five stages. None are “just call an API.” And the real engineering challenge is making them all complete in under one second.

Core Challenge #1: The Latency Budget

“Realtime” isn’t an adjective — it’s a quantifiable budget. End-to-end latency under 1 second is the standard for live captions. Here’s how that second gets split:

Stage	Typical Latency	Notes
Capture + buffering	20–100 ms	Audio is gathered in small chunks
Noise suppression	5–20 ms	RNNoise runs in realtime
VAD	5–30 ms	Silero VAD per-frame is sub-millisecond
ASR inference	100–500 ms	The biggest variable — model and device dependent
Network round-trip (cloud)	50–300 ms	Cloud-only cost
Diarization	100 ms – several seconds	The most unpredictable factor

Once the total exceeds 1 second, the realtime feel disappears. Architecture selection is fundamentally about which items in this table you’re willing to pay for.

Core Challenge #2: Streaming vs Batch — a Fundamental Divide

• Batch: Wait for the full recording, then transcribe all at once. Highest accuracy (full context available), but not realtime.
• Streaming: Output text as audio arrives. Must emit partial (tentative, changes as more context arrives) then final (committed).

Here’s the key insight: powerful models like Whisper are inherently batch models. They aren’t designed to stream. Making them “realtime” requires engineering tricks — sliding windows, overlapping chunks, incremental decoding — to simulate streaming behavior. This is the central difficulty of the on-device approach.

Core Challenge #3: Streaming Diarization Is Near-Research-Grade

This is the single most important thing to understand:

Streaming speaker diarization (identifying “who’s talking” in real time) is borderline unsolved.

Diarization relies on voiceprint clustering — extracting speaker embeddings and grouping similar voices. Clustering inherently needs enough samples to be accurate. In streaming:

• At the start of a sentence, the model has almost no voiceprint data for that speaker → guesses are unreliable
• As more audio arrives, the model gets more confident → it may retroactively change earlier labels (“That was actually Speaker B, not A”)
• Overlapping speech (two people talking at once) is nearly impossible

In practice, every “realtime diarization” system either lags speaker labels by 1–2 seconds, or guesses first and corrects later. Understanding this upfront is essential — there’s no free lunch, only the choice of whether to tackle this on-device or in the cloud.

Architecture 1: Cloud Streaming First

Device: capture → noise suppression → VAD → audio stream (WebSocket)
                                            │
                                            ▼
Cloud streaming API (Deepgram / AssemblyAI / Azure Speech)
   └─ streaming ASR + streaming diarization + auto language detection

The device does minimal preprocessing. Audio is pushed to the cloud over WebSocket. The cloud returns transcribed text + speaker labels + language info in real time.

Why it’s strong:

• Natively supports streaming — protocols designed for partial/final incremental output, <300ms for first draft
• Built-in diarization — you never touch speaker embedding clustering
• Multilingual with auto-detection — one endpoint covers dozens of languages
• Continuously improving — models update server-side, no app update needed

The trade-offs:

• Per-minute cost — cheaper services run a few cents per minute. Fine for occasional use, but if users leave live captions on for hours, costs scale linearly.
• Network-dependent — no connection, no captions. Weak connection, delayed or missing words. Needs a degradation strategy.
• Privacy is the sticking point — audio uploads to third-party servers. GDPR, data residency, user consent — non-negotiable for a recording app sold internationally.
• Vendor lock-in — every API has different protocols, result formats, and diarization semantics. Wrap them behind a unified ASR abstraction layer.

When to choose it: You want the fastest path to a working, accurate version. Users don’t particularly care about audio going to the cloud. You don’t want to build diarization yourself.

Architecture 2: On-Device Whisper

Device (everything local, no network):
capture → noise suppression (RNNoise) → VAD (Silero)
   → whisper.cpp sliding-window pseudo-streaming
      → local speaker embedding + online clustering for diarization

Everything runs on the user’s machine. The core is whisper.cpp — OpenAI Whisper’s C/C++ port, optimized for consumer hardware. On Apple Silicon it uses Metal acceleration; on Intel, it runs on the CPU.

Making a batch model stream:

Whisper is batch by nature: it ingests 30 seconds of audio and decodes in one pass. The engineering tricks to fake streaming:

• Sliding window with overlap — every 0.5–1 second, feed the most recent N seconds of audio for a new decode pass. Overlapping windows prevent words from being cut off at chunk boundaries.
• Partial/final dual output — the latest window’s result is partial (may change). When VAD confirms a sentence is complete, lock it as final.
• VAD-driven segmentation — use Silero VAD to chop long audio into sentences. Smaller chunks = less computation per pass = lower latency.
• Model tiering — tiny/base for speed, large-v3 for accuracy. Common pattern: small model for drafts, large model for finals.

On-device diarization is the hard part:

Cloud services give you diarization for free. On-device, you build it yourself:

1. VAD segments speech chunks
2. A speaker embedding model from a diarization toolkit (pyannote.audio, 3D-Speaker, WeSpeaker) extracts voiceprint vectors per chunk
3. Online clustering groups similar embeddings → same speaker
4. Streaming requires incremental clustering updates, handling new speaker arrivals and retroactive label changes

This is heavy engineering. Many on-device projects ship ASR first and postpone diarization or degrade it to “compute after recording stops.”

The value:

• Zero marginal cost — no per-minute charges
• Maximum privacy — audio never leaves the device. This is a genuine differentiator and marketing asset for a recording app.
• Works offline — airplanes, basements, anywhere
• Full multilingual capability — Whisper large-v3 handles dozens of languages

The costs:

• CPU/memory/battery — medium-to-large model inference on older hardware is punishing
• Sub-1s latency requires heavy tuning — window size, overlap, model selection all interact
• Diarization is hard and heavy
• Model distribution — large models are gigabytes, bloating app size

When to choose it: Privacy and offline use are core product differentiators. You’re willing to invest engineering effort. Users have relatively modern hardware (Apple Silicon is ideal).

Architecture 3: True Hybrid (Edge + Cloud)

         ┌────────────── On-Device ──────────────┐
capture → noise → VAD ─┬→ Apple Speech (device streaming)
                       │     → partial "instant draft" (0 latency, gray)
                       │
                       └→ same audio → cloud streaming API (parallel)
                             │
                             ▼
                  final "high-accuracy" output + speaker labels
                             │
                             ▼
          Subtitle state machine: replace device partial
          with cloud final, backfill speaker tags

The core idea: device handles speed, cloud handles accuracy. Each side does what it’s best at.

Why hybrid is the only approach that satisfies both <1s latency and diarization:

• Pure cloud: network round-trip eats into the budget, weak/no network = dead
• Pure device: large model inference strains hardware, diarization is a beast
• Hybrid splits the tension:
  1. Apple Speech (SFSpeechRecognizer) supports device-side streaming — free, offline, near-zero latency (requires a one-time model download; utterances capped at ~1 minute). It emits the first draft instantly → the user sees text immediately
  2. Cloud streaming API processes the same audio in parallel, returning high-accuracy results + speaker labels a few hundred ms later → these replace the draft and add speaker tags
  3. Offline fallback — when the cloud is unavailable, device drafts promote directly to final. Function never breaks.

The real complexity: the subtitle state machine

The hardest part of the hybrid approach isn’t the recognition — it’s managing the lifecycle of every subtitle line:

[device partial · gray]
  → [device final · black]
    → [cloud final · more accurate black text]
      → [speaker label appended · "A: ..."]

You need a robust state machine to handle:

• Time alignment — the same utterance has slightly different timestamps from the two engines
• Replacement timing — when to overwrite device output with cloud output, without the on-screen text flickering or jumping
• Speaker backfill — diarization corrections (retroactive label changes) must update the UI smoothly
• Degradation switching — seamless transitions between “pure device” and “hybrid” as network quality fluctuates

The trade-offs:

• Two engines, two result streams to merge — highest overall complexity
• “Draft then revise” UX must be carefully designed, or users perceive the captions as “jittery”
• Cloud costs and privacy concerns remain (mitigated: cloud is optional, on only when connected and user consents)

When to choose it: You want low latency, high accuracy, AND offline resilience simultaneously. You’re willing to invest in best-in-class UX. Your product positioning is explicitly “edge-cloud hybrid.”

Side-by-Side Comparison

Dimension	Cloud Streaming	On-Device Whisper	Hybrid Edge-Cloud
End-to-end latency	Low (~300ms)	Medium, tuning-dependent	Lowest (device draft instant)
Accuracy	High	High (large-v3)	High (cloud refinement)
Speaker diarization	Built-in, easiest	Self-built, hardest	Cloud handles it
Multilingual	Strong	Strong	Strong
Works offline	❌	✅	✅ (device fallback)
Privacy	Weak (audio to cloud)	Strongest	Medium (cloud optional)
Cost	Per-minute	Zero marginal	Medium (cloud portion)
Engineering effort	Lowest	High	Highest
Device load	Light	Heavy (CPU/battery)	Medium
Best for	Speed to market, accuracy-first	Privacy/offline as differentiator	UX-first, edge-cloud positioning

How to Choose

There’s no “best” architecture — only the one that matches your product stage:

• Fastest validation, accuracy priority → Architecture 1. Get the feature working with cloud APIs first, collect user feedback, let mature services handle diarization and multilingual.
• Privacy and offline are core differentiators, willing to invest → Architecture 2. This is the path that builds the strongest moat — especially as “audio never leaves your device” becomes more valued by users.
• Best UX, aligned with edge-cloud philosophy → Architecture 3. The best balance of latency, accuracy, and offline resilience. Highest engineering cost (especially the subtitle state machine).

Ask yourself three questions: Do my users care about privacy? Do they have reliable internet? How much engineering am I willing to invest? The answers will point you to one of these paths.

---