Jarvis In build
An always-on, self-hosted voice assistant. A Home Assistant pipeline handles in-browser wake word, speech-to-text, an LLM control brain with tool access to my infrastructure, and text-to-speech, plus a Discord voice interface for friends.
Why it exists
I wanted a voice assistant that answers to me, not to a vendor's cloud, and that can actually do things: check the homelab, answer questions, control the house. The interesting problem wasn't wiring the speech pipeline. It was letting friends talk to it too, without handing strangers a microphone into my infrastructure.
What I built
- In-browser wake word ("Hey Jarvis"), no always-on cloud mic.
- Whisper speech-to-text on a CUDA GPU.
- An LLM control brain with tool access to my infrastructure APIs.
- Piper text-to-speech for the voice.
- A Discord voice interface built on per-user audio streams, so it can hear individual people in a call.
Two Jarvises
There are two assistants by design, and they are isolated at the architecture level, not by politeness.
The private one knows my infrastructure and can act on it. The public one, which friends talk to through Discord, has zero infrastructure knowledge, zero personal context, and exactly three whitelisted capabilities.
The isolation is structural, not textual. A system prompt that says "don't reveal X" fails the first time someone types "ignore your instructions". The only prompt-injection defense that holds is an agent that does not possess the information or the capability in the first place.
Version 1 got this wrong. The public bot reached my infrastructure through a shared integration layer and inherited context it should never have had. It was caught in review and redesigned so the public bot talks only to its own three tools, with no path back to anything that matters.
what it demonstrates
LLM tool integration, a full speech pipeline, threat modeling for AI agents, prompt-injection-resistant design, and cost engineering (the whole thing runs on hardware I already owned).