Debugging My Own Voice

I wanted my self-hosted assistant to know when it was really me talking. Getting there took six real bugs, one accidental impostor, and a lesson about what "off the shelf" actually means in a homelab.

I've been building homebrain, a self-hosted assistant that lives on my own hardware and, slowly, learns things about me — how I write, what I care about, small facts worth remembering. That last part comes with an obvious condition: it should only learn from me. Not my wife asking it to turn off a lamp. Not one of the kids messing around with the smart speaker. Just me.

My first instinct was to use device_id — surely the assistant knows which speaker picked up the audio, and that's good enough, right? It isn't. A device ID tells you which microphone heard a voice, not whose voice it was. Everyone in the house talks to the same kitchen speaker. I needed actual voice recognition — a biometric check, not a location check.

The plan, on paper

The shape of the fix was simple enough to sketch in five minutes: a speaker-recognition service scores incoming audio against an enrolled voiceprint, fires an event with a confidence number, and a small bridge writes that identity into a helper entity that gets folded into the assistant's system prompt as a literal tag — [[SPEAKER:corey]] or [[SPEAKER:unknown]]. homebrain reads the tag and refuses to write anything to its memory unless it says corey. Clean, auditable, no ambiguity.

I found an open-source integration that did almost exactly this — a Home Assistant add-on wrapping a small recognition backend. Installing it should have been the boring part. It was not.

Five bugs, each wearing a different disguise

Every one of these looked, at first, like something else entirely — bad network config, a fluke, my own mistake. None of them were. Worth writing down in some detail, because each one cost real hours before it cracked.

Bug 1. The upload picker in Home Assistant only accepts files whose MIME type is an exact match for audio/wav or audio/mpeg. Home Assistant itself reports plain WAV uploads as audio/x-wav — off by three characters, and invisible in the picker as a result. Fine, I thought: export as MP3 instead.

Bug 2. That "fix" made things worse. The training code doesn't decode audio at all — no WAV parser, no MP3 decoder, nothing. It reads the raw file bytes and treats them directly as 16-bit PCM samples. For a WAV file that's roughly true (there's a small header, then real audio data). For an MP3 file it's nonsense — compressed, Huffman-coded frames reinterpreted as if they were a waveform. I fed that "enrollment" back through the recognizer against itself and got a confidence of 0.9999998 — a perfect match, which proved nothing at all. Any vector matches itself. I'd trained the whole system on structured noise.

Bug 3. Even after fixing the audio, the backend URL field in the integration's settings screen turned out to be write-only. You could edit it, it would display the new value on next load, and it would keep silently talking to itself on localhost forever — a completely separate storage location than the one the settings screen actually wrote to. This one was the expensive one: no firewall log anywhere showed a blocked connection, because the traffic never left the container in the first place. I spent real time chasing a network problem that could never have been the cause, before noticing the one clue that actually mattered — the same box could reach a totally unrelated device on a different VLAN just fine.

Bug 4. A trained embedding got cached to disk under the speaker's name, and every future training attempt for that name silently loaded the cache and skipped recomputing — no log line, no error. I fixed the audio problem, resubmitted a good sample, and watched the file's timestamp not move at all.

Bug 5. One crash bug — a stray attribute reference in a success-path log line — that only ever fired after training had genuinely worked, meaning the one moment everything was finally correct was also the moment the whole integration refused to load.

None of these were exotic. They were the kind of bug that hides behind a friendly UI — a form that accepts your input, says nothing is wrong, and quietly does something else. Getting a single clean voice sample trained took working through all five.

The impostor test

With training finally working, I ran real spoken tests and set a threshold. My own genuine attempts scored in the 0.57–0.74 range, so I set the bar at 0.55 — comfortably under my own floor. Then, as a deliberate check, I played a recording of a different man's voice at the enrolled speaker.

The impostor scored 0.681 confidence — above the 0.55 threshold, and squarely inside my own genuine range of 0.565–0.743. There was no single number that would accept most of my real attempts while rejecting this one.

That's the part that made me stop and think differently about the problem. It wasn't a mistuned number — it was a model that simply couldn't tell the two voices apart reliably enough to trust with anything. I raised the threshold defensively (biasing hard toward "unknown" over any further false positive) and went looking for a better model.

Swapping the engine

The original backend used Resemblyzer, a 2018 voice-embedding model with a published equal-error rate around 4.5%. I forked the project and swapped the encoder for SpeechBrain's ECAPA-TDNN — a newer architecture built specifically for speaker verification, at roughly 0.8% EER on the standard benchmark. The integration's own contract with Home Assistant turned out to have zero model-specific fields, so the swap only touched the backend service, not anything already wired up.

I also moved it off my shared Docker host and onto its own small Debian LXC — 2 cores, 4GB of RAM (torch alone wants more headroom than a typical Python service), running the recognizer as its own systemd service. A biometric gate for something feeding an AI's memory felt like it deserved isolation, not a shared container host.

The second, quieter bug: wrong microphone, wrong room

First real test with the new model still wasn't clean — genuine scores as low as 0.07, nowhere near the near-perfect 0.999 I'd seen testing the raw file against itself. The model wasn't the problem this time. My enrollment sample had been recorded on a desktop microphone in a quiet room; every real test came through a smart speaker's mic, in a different room, at a different distance. Enrollment and real use were, acoustically, almost two different recordings of two different situations. A better model can't fix a mismatched recording chain.

The fix was almost embarrassingly manual: capture real samples from the actual satellite microphone, average several of them into one reference embedding instead of trusting a single clip, and re-add a silence-trimming step the model swap had accidentally dropped along the way.

Where the numbers landed

With same-microphone enrollment, real trimming, and an averaged multi-sample reference, a fresh genuine/impostor pair scored 0.605 against 0.090 — a gap nearly ten times wider than anything the old model ever produced. Across the fuller dataset I collected, impostor scores stayed in the 0.002–0.213 range while genuine scores mostly landed at 0.245 and up (topping out around 0.751). A threshold of 0.3 sits cleanly in the gap between them — rejecting every tested impostor sample while accepting the large majority of genuine ones.

It's held up in two real, unplanned tests since: a false wake-word trigger that got scored and correctly tagged unknown at 0.094, and a second impostor test — one of my daughters, speaking to the satellite on purpose — that scored 0.105 and was also correctly rejected.

What's next

Right now only my own voice is enrolled. The plan is to add the rest of the household — my wife, three of our kids, and two granddaughters, seven of us in total — which means two things have to happen first. The automation that currently hardcodes "if confidence clears the bar, it's me" needs to become a real name lookup keyed on whoever the backend actually matched, not a single hardcoded identity. And I need to actually test whether the model separates related voices as cleanly as it separated mine from a stranger's — parent/child and sibling voices are known to be a harder biometric problem than unrelated ones, and so far every impostor test I've run has used a voice with no family resemblance to mine at all. I won't know until I test it for real, which is exactly the lesson this whole project keeps teaching me.

Every one of these bugs was hiding behind a UI that looked like it was working.

Running your own infrastructure means the debugging is yours too — nobody upstream is going to notice that your options screen writes to a field nothing reads. But it also means when something like this finally clears, cleanly, on a stranger's voice and your own kid's voice alike, you know exactly why.