Runs entirely on-device: speech recognition, reasoning, and voice synthesis all happen locally, on local models, on your own hardware. No cloud API in the loop, no subscription, no data leaving the machine.
IRA is a fully local "Jarvis" I designed and built solo: say "hey jarvis," and it listens, talks back in a natural voice, and actually controls your desktop, all without a single request leaving your machine.
The Problem
Alexa, Siri, and Google Assistant are the products everyone compares this to, and none of them do what this needed to do:
- They're cloud-bound: every word you say leaves your machine
- They can't touch your desktop: no window management, no app control, no "read my clipboard"
- They're generic: no persistent identity, no memory of who you are or what you've told them before
- They can't be interrupted gracefully: most voice assistants either ignore you mid-response or need a hard stop-word
I wanted an assistant that felt less like a smart speaker and more like a person sitting next to my desktop. Private by construction, able to actually do things, and able to hold a real conversation instead of a scripted one.
What It Does
| Capability | What it looks like in practice |
|---|---|
| Wake word + streaming speech | "Hey Jarvis" opens the mic. Partial transcripts stream in real time, with an async correction pass cleaning up the final text. |
| Natural conversation | Free-form chat, jokes, questions, personal context, routed through a local LLM with a real personality, not a canned command grammar. |
| Desktop control | Volume, brightness, Wi-Fi/Bluetooth toggle, app launching, window snap/minimize/maximize, media playback, clipboard read/write, countdown timers. |
| "Show me" mode | Instead of silently changing a setting, it walks you to it on screen and explains what to click. |
| Vision | Takes a screenshot and describes what's on screen, or streams a live phone-camera feed over the network for "look at this" moments and face recognition. |
| Memory | Remembers notes you ask it to save, and recalls them later. A first step toward a genuinely personal assistant. |
| Real barge-in | Interrupt it mid-sentence and it actually stops and listens. The hardest part of the build, covered below. |
Under the Hood
Everything is streamed end to end, so the assistant starts talking before it has even finished "thinking":
- Speech recognition: a real-time Whisper pipeline that emits partial transcripts as you speak, plus a background correction pass on a larger model, fast enough to feel instant, accurate enough to trust.
- Understanding: a local LLM (Ollama) classifies every utterance into a structured action (
chat,volume,window,web_search,guide,timer,screen_capture, and more) and produces the exact sentence to speak back, all in one pass, as strict JSON. - Voice: GPU-accelerated TTS speaks each sentence as soon as the LLM produces it, instead of waiting for the whole response, shaving real seconds off perceived latency.
- Vision: screenshots and a phone-camera feed, tunneled securely over the private network, are handed to a local vision model for description and face recognition.
- Everything is instrumented: every turn logs a stage-by-stage latency breakdown (listening → first LLM token → first spoken word), so "it feels laggy" is a measurable regression, not a vibe.
The Hardest Bug, and Why It Mattered
The most interesting engineering problem wasn't the happy path. It was making the assistant not talk over itself.
Fixing a real audio bug (Bluetooth output going silent) required turning off the browser's built-in echo cancellation. The side effect: the assistant started hearing its own voice through the microphone mid-response, and occasionally mistook its own words for a new command, or worse, cut its own sentence off mid-word because it thought it was being interrupted.
The fix took three iterations, each one peeling back a deeper layer of the problem:
- Reject an echoed transcript after the fact. Worked for the obvious case, missed a subtler leak path where trailing audio bled into the next listening window.
- Reject any echoed transcript, from any path. Closed the false-command bug completely, but the assistant was still audibly cutting itself off mid-sentence before the check could even run.
- Verify before cutting. The real fix: instead of hard-muting the moment it thinks it hears you, it keeps talking, quietly checks what it's hearing against speech-to-text in the background, and only actually interrupts itself once that check confirms it's really you, not an echo of its own voice. The trade-off is a few hundred milliseconds of confirmation delay on a genuine interruption, in exchange for zero false self-interruptions.
That progression, treat the symptom, then find the real path, then redesign the interaction instead of patching around it, is the kind of debugging this project demanded constantly. A voice interface has no room to fake it: every bug is instantly, audibly obvious.
Engineering Principles
- Local-first by default: every design decision defaulted to "can this run on-device" before reaching for a network call.
- Streaming over batching everywhere it mattered: recognition, reasoning, and speech all overlap instead of running in sequence, because perceived latency is the entire user experience of a voice product.
- Instrument before you optimize: per-turn latency logging exists specifically so performance claims are backed by data, not impressions.
- Fix root causes, not symptoms: the barge-in saga above is the clearest example. Two "working" fixes were shipped and superseded before the actual architectural fix was found.
What's Next
- Ambient, always-on vision instead of only responding when asked to look
- A configurable persona/identity instead of a single hardcoded personality, the groundwork for a multi-user or shareable version
- A packaged, installable build (currently runs from a dev checkout)
- Structured, queryable long-term memory instead of a flat notes list
- A tightened wake-to-response pipeline using onset detection that's already available server-side but not yet fully wired into the client
Status
In active daily use and development. Built solo, end to end: real-time audio pipeline, local LLM orchestration, GPU voice synthesis, desktop automation, and a from-scratch barge-in/echo architecture, all running on local hardware with no cloud dependency in the core loop.
Curious how the streaming pipeline or the barge-in fix works, or want something like this built for your product? Get in touch.