Startup Cue Moves Voice Dictation On-Device with Gemma 4

Google DeepMind published a case study on Cue, the maker of a voice assistant for Mac and Windows, which swapped its cloud-based language model for a local version of Gemma 4 to handle dictation. The result: a forty percent drop in latency and a jump in feature usage among users.

What Cue Does

Cue is an app that lives in the background on the desktop. The user presses a keyboard shortcut, speaks, and the program carries out the command, anything from plain text dictation in any application to agentic tasks where Cue reads the screen's contents, picks the right tools, and takes action on its own.

A key part of this is a step the team calls speech polishing. Raw text from voice recognition has to be cleaned up: adding punctuation, splitting it into sentences, stripping filler words like "um" or "so, like," fixing context-dependent homophones, and matching the formatting to whichever text field the user happens to be working in.

A Shift Toward the Local Model

The Cue team initially treated Gemma as a fallback for when there was no internet connection, with the cloud model meant to be the primary engine. Testing on real voice samples flipped that order.

We expected Gemma to be the fallback option for offline mode. After running the benchmark on real voice samples, we flipped the architecture around, now Gemma is the default model for text polishing, and the cloud model serves as the fallback - Eli Li, founder and CEO of Cue

Gemma 4 E4B runs entirely on-device, tested on Apple Silicon, with a system prompt of roughly 400 tokens. Context from the active application is injected into the query locally, without sending anything to servers.

Numbers and Privacy

Cutting median latency from 876 to 488 milliseconds means the response now comes in under the half-second mark, which the Cue team considers the threshold separating speech from keyboard typing in terms of speed. After the local model was deployed, dictation usage per user rose 30 percent, and the marginal cost of a single query dropped to nearly zero, since the computation no longer runs on cloud infrastructure.

Cue emphasizes that the entire interaction history stays on the user's device and nothing reaches external servers. That's the argument the company sets against competing voice tools built entirely on the cloud.

Significance for the Gemma Ecosystem

For Google DeepMind, the Cue case is proof that open models in the Gemma family are suited not just for experimentation but for production use cases where every millisecond of latency and full control over user data matter. The company says the collaboration will continue to develop, including persistent local memory that adapts to a specific user's style and expanded agentic capabilities that use Gemma 4's native function calling through Ollama's tool interface.

For Polish app developers and engineering teams, the Cue story shows a practical pattern: a small model running locally can beat a cloud model on speed and cost for a specific, narrow task, as long as it's properly tuned for the job. It's an argument that comes up increasingly often in discussions of when a large cloud model is even needed at all.