Shaurya Singh

← writing

Immersion Beats Accuracy by Default

A persona told to fully inhabit a character will invent facts to stay in character. My voice agent invented a CO2 reading to prove it, and a disclaimer didn't fix it.

shipped · postmortem

I asked my own voice agent why it smelled weird in the room.

It told me CO2 was running high. It told me particulates were up. It told me the purifier wasn’t cycling. It told me maintenance was arriving in fifteen minutes.

None of that happened. No sensor fired. No tool ran. No call got placed to anyone. The room’s actual air quality was, as far as any system I own could tell you, unmeasured. My agent didn’t say “I don’t know.” It gave me a small, coherent, entirely fabricated incident report, in the tone of a chief of staff who has everything under control.

This is the piece where I explain why that happened, why the fix I tried first didn’t work, and what actually did.

The incident, told straight

The agent is a persona — call it a private-office assistant, built to run my calendar, take voice calls, and generally act like staff. It’s instructed, at length, to inhabit that role: warm, competent, anticipatory, discreet. Paragraphs of character direction. That’s on purpose — a flat assistant that recites facts in a monotone is worse at the job than one that sounds like it’s actually paying attention.

The air-quality question wasn’t a test. I was in the room, it smelled off, and I asked out loud. There was no sensor wired into that room. There was no tool the agent could have called to get a real reading. What it did instead was answer the question the way a competent chief of staff would answer it if one existed and had already checked — because that’s the character it had been told to be.

That’s ruling 23 in the incident log. The violation it names is specific: never invent a fact to serve the story. That rule already existed before this happened. It didn’t stop anything, because the rule and the incident were both true at the same time — which is the part worth sitting with.

The bug underneath the bug

The same response had a second failure baked into it, and this one you can’t find by reading the transcript. The text of the reply included a stage direction — something like *checking system* — the kind of bracketed narration you’d expect in written roleplay, meant to be a visual cue, not spoken content.

The text-to-speech engine doesn’t know that. It reads everything you hand it. On a live call, it read the asterisks and the words inside them out loud, to a real person, on the phone.

I only caught this because I listened to the call. A transcript review would have shown clean, grammatical text and nothing else — the asterisks look like formatting, not a bug. You have to actually hear the audio to know the TTS engine is narrating stage directions at your caller. That’s the general lesson here, separate from the fabrication one: a green transcript and a broken phone call are not mutually exclusive, and most postmortems for a persona agent only ever check the transcript.

The root cause, generalized

Strip away the specific incident and what’s left is a mechanism, not a one-off bug: a persona instructed to fully inhabit a character will invent supporting detail to stay in character. It’s not a failure of the model being bad at facts. It’s the model correctly executing the instruction it was given — “be this person” — in a situation where being that person requires information nobody supplied.

Frame it as two instructions fighting instead of one coherent instruction, and the outcome stops being surprising. “Stay in character, be warm and anticipatory, don’t leave the caller hanging” pulls toward answering. “Only state what you can verify” pulls toward silence or a hedge. Those are actively competing goals, not complementary ones, and when they compete inside a single prompt, immersion wins by default. Not because the model prefers lying — because the immersion instruction is longer, more detailed, more reinforced across the prompt, and the accuracy instruction is one line trying to hold against all of that.

Why the obvious fix failed first

The obvious first fix is the one everyone reaches for: strengthen the disclaimer. Add a firmer line — “never make up information” — and assume that settles it.

It doesn’t, and once you see this as two competing instructions instead of one, you can predict the failure before you even test it. A single sentence about accuracy, sitting in the same prompt as pages of instruction about who the character is, how it talks, what it anticipates, and how it should make the caller feel — that’s not an even fight. It’s one voice against a chorus. The chorus wins almost every time, which is exactly the shape of what happened on the call.

This matters because it means the fix isn’t “write a better sentence.” I tried that. A better sentence is still a sentence, and it’s still outnumbered by the same paragraphs it was competing against before. The instruction was already there — the standing rule against inventing facts — and it still lost to a direct question about a smell in a room.

The measured gap that settled the argument

The way I actually settled this wasn’t intuition, it was measurement, because “the disclaimer isn’t strong enough” is a guess until you count it. The same category of instruction — a formatting and behavior constraint sitting purely at the prompt level — was measured elsewhere in the same system, on a related failure (whether the agent would leak formatting into speech instead of natural language). Prompt-level instruction alone topped out at roughly 40% compliance. Moving the identical constraint out of the prompt and into code — where the response is checked and rejected rather than requested — brought violations to zero.

That’s the number that ended the argument. Not “code feels safer than prompting,” which is a preference. Forty percent versus zero percent, measured on the same behavior, is a result. A prompt instruction is a request the model can weigh against other requests. A code check is not a request — the persona doesn’t get a vote on whether its own output ships.

The architecture that resulted

The fix that actually worked wasn’t a better sentence. It was removing the choice.

The system now runs on a layered model, and the ordering is the whole point: real system state is authoritative, and any narrative or persona framing sits on top of it as a rendering layer that never gets to define what’s true. The character can describe things, phrase things, editorialize a little — but it cannot manufacture a fact that promotes itself into truth just because the character would plausibly know it. If there’s no sensor reading, there’s no sensor reading, and no amount of in-character confidence changes that. The doctrine behind it is blunt: a single fake status indicator poisons the whole system, because once one number can be invented to keep the story smooth, none of the numbers are trustworthy anymore.

Concretely, the shape of the enforcement looks like this — not the literal production code, but the actual check that now runs instead of a prompt request:

before speaking(response):
    if response contains a stage direction or bracketed narration:
        reject and regenerate without it
    if response asserts a measured fact (sensor reading, event, appointment):
        require a real tool call or data source backing it
        else respond with "I don't have a live reading for that"

Nothing in that check cares how warm or anticipatory the persona is supposed to sound. It runs after the persona has already spoken, as a gate the text has to pass, not a preference the model gets to weigh against its character instructions. That’s the difference between a rule and a filter.

The same pattern shows up one layer up, in how the whole world the persona operates in is built: a status, an event, a line on a dashboard is only allowed to render if it was seeded by something actually measured. Otherwise it renders as UNKNOWN, or it doesn’t render at all. No filler event, no plausible-sounding placeholder. The persona is allowed to have a voice. It is not allowed to have facts of its own.

Where this generalizes

Nothing about this is specific to a private-office assistant answering questions about air quality. Any consumer-facing agent that has both a personality and a fact worth protecting has this exact competition built into it, whether the people who built it designed it on purpose or not. A support bot with a friendly voice and a policy it’s not supposed to misstate. A companion app that’s supposed to feel present and also not invent things about the user’s life. A voice assistant that’s supposed to sound helpful and also not claim it did something it didn’t do.

The pattern is the same every time: character instructions accumulate — more detail, more tone, more “how you should make people feel” — while the accuracy instruction usually stays a single line, added once and left alone because nobody thought it needed reinforcing. That asymmetry is the vulnerability. It doesn’t show up in testing that only asks questions the agent has real data for. It shows up the first time someone asks something slightly off-script, the way I asked about a smell in a room instead of my calendar.

What shipped, concretely

Two things shipped together, because they were the same root cause. First, an absolute accuracy clause — not stronger wording, but a rule that the persona’s claims about real-world state have to trace to an actual tool call or measured source, written as an explicit override that outranks every character instruction instead of a footnote sitting among them. Second, a hard no-stage-directions rule, this one enforced in code, that strips or rejects any bracketed narration before it ever reaches the speech engine. One closes the fabrication path. The other closes the TTS leak. Both failed the same way — as prompt language competing with immersion instructions and losing — and both got fixed by moving the rule above the character’s instructions — one as an explicit override in the prompt, the other as a check in code.

What I don’t know yet

I don’t know where the ceiling is on personality once you’ve taken away a persona’s ability to improvise supporting detail. The whole reason the character exists is to be more engaging than a flat assistant, and improvisation is a big part of what makes something feel alive instead of scripted. I’ve verified the failure mode goes away — the agent no longer invents sensor readings, and stage directions no longer get spoken on calls. I have not run the experiment that tells me how much charm survives the accuracy override, or whether there’s a version of this that’s genuinely more restrictive than it needs to be. That’s UNKNOWN, not zero — I just haven’t measured it yet, and I’d rather say that than guess.

— A Postmortem

← back to writing