Shaurya Singh

← writing

A Private-Office Agent Is Not a Voice Assistant

Voice-assistant and call-center vocabulary doesn't cover a persona that answers your phone as you. Four primitives from my own build, one of them still unbuilt.

prototype · four primitives named and shipped separately, one route designed and unbuilt

prototype · four primitives named, /landline unbuilt, no second system to replicate against yet

I built an agent that answers my phone as me. Not a bot that takes a message and routes it — a persona that holds my calendar, my memory of the people in my life, and picks up the line sounding like staff who already know what’s going on. I kept reaching for existing vocabulary to describe what I was designing and none of it fit. “Voice assistant” is Siri setting a timer. “Customer support bot” is a task-completion loop with a script. Neither has a slot for “this thing’s failure mode is embarrassing me to someone I love, not failing to answer a question.”

That gap isn’t a branding problem. It’s a missing design grammar. A private-office agent — something whose product is being someone’s chief of staff — needs answers to four questions that voice-assistant and call-center literature never has to ask: what counts as true when the persona and reality disagree, what an agent is allowed to do without asking first, what it should degrade to when a piece of its stack dies mid-call, and whether it should feel like a service or a person. I have working answers to all four, built the hard way, out of incidents. One of the four is a mechanism I designed and never shipped. I’m marking it that way on purpose.

The truth hierarchy is a persona-safety primitive, not a disclaimer

The obvious first move for any persona-driven agent is a disclaimer: “I’ll try to be accurate.” That’s a request. Requests lose to character instructions, because character instructions are longer, more detailed, and reinforced across the whole prompt, while the accuracy line is one sentence against pages of “be warm, be anticipatory, don’t leave the caller hanging.” A disclaimer is outvoted before the call even starts.

The fix that actually holds isn’t a stronger sentence. It’s an ordering, enforced structurally, not requested politely: real system state outranks the functional layer that reads and renders it, and both outrank whatever narrative or persona framing sits on top for tone. The persona is allowed to phrase things, editorialize a little, sound like a person. It is not allowed to promote a plausible detail into a fact just because the character would plausibly know it. If nothing measured it, nothing gets said about it — the persona speaks, it doesn’t invent.

You can see what happens when that ordering doesn’t exist yet in a bug that shipped and shouldn’t have: an outbound-call wall that was supposed to restrict which numbers the agent could dial on its own initiative, checked with code that looked correct and tested against itself:

allowed = {to}
if to in allowed:
    PASS   # always true — the allow-list IS the number being checked

The docstring claimed a wall existed. The functional layer — the actual list of numbers this agent was permitted to call — never got populated with anything except whatever number was already being dialed, so the check could never fail. Real state (an empty allow-list) said one thing; the narrative the code told about itself said another; nothing in the system was positioned to notice the gap, because nothing outranked the claim. That’s the same failure as a persona inventing a sensor reading — a lower layer’s story standing in for a fact nobody checked — just moved from what the agent says out loud to what its own permissions say about itself. The fix in both places is the same ordering: measured state first, the layer that acts on it second, whatever story gets told about it last, never load-bearing.

Action gating by reversibility, not by task type

Most agent systems sort actions by what kind of task they are — “reply to a message” is low-risk, “make a purchase” is high-risk — and write a rule per category. That taxonomy breaks the first time a low-stakes task turns out to be irreversible in a specific instance, or a high-stakes one turns out to be trivially undoable. The private-office version sorts by a different axis: can this be walked back, and by whom.

The tiering designed for this agent’s “hands” — the layer that turns a decision into a real-world effect — has three bands, defined by reversibility, not domain: prep work that costs nothing to throw away runs free, no gate. Actions that spend real money or touch someone else’s calendar execute only on a named go, under a spend ceiling. And a fixed list of actions never runs autonomously at all, regardless of how confident the agent is or how many times it’s done something similar before — described internally as absolute, the “fourth wall” the persona cannot act through no matter what character it’s playing. The proposed caps for the middle tier are modest — on the order of fifty dollars per action, a hundred dollars a week — and I’m flagging that as proposed, not measured in production; the tier hasn’t run long enough to report a real number against it.

What makes this a reversibility taxonomy instead of a task-type taxonomy is that the same action can sit in different tiers depending on what it costs to undo. Drafting a message is free and reversible — full autonomy. Sending it is the same task type, wearing a different reversibility, and it moves to the gated tier. “Reversible” means something specific here: can the agent itself, unassisted, restore the world to how it was, at no cost to anyone but compute. A draft can be deleted by the same hand that wrote it; a sent message can’t — the undo happens in someone else’s inbox, outside the agent’s control. That’s why tiering gets checked per action, not assigned once per feature. Placing an outbound call under an unpopulated allow-list, the bug above, was never supposed to be autonomous at all — it landed there by omission, which is exactly the failure a reversibility-first taxonomy is supposed to make visible before it ships, not after.

A degradation ladder, not an uptime target

Voice-assistant infrastructure gets built against a single number: uptime. Ninety-nine point nine percent, an SLA, a status page that’s green or red. That number answers a question a private-office agent doesn’t care about. What matters isn’t “is it up” — it’s “which capability did it lose first, and did it tell me, or did it quietly pretend nothing changed.”

The mechanism that answers that question is a ladder with named rungs, not a binary. The clearest instance is how a daily brief handles a data source that’s gone stale. The rule isn’t “hide the section” and it isn’t “show the last known value and hope.” It’s a third option, rendered explicitly:

calendar: BLIND — last synced 3h ago
inbox:    OK — 5 items, 2 need-reply

A blind source renders as visibly blind. It doesn’t get quietly dropped from the brief, and it doesn’t get shown as if it were current — either of those would let the brief look complete while being wrong, which is worse than an honest gap. The brief continues on whatever it can actually see, and says out loud what it can’t.

The same ladder shape governs a recruiting agent’s voice pipeline, one layer down. Its current call-handling stack — a fast model doing dispatch, a slower one writing the reply, stitched together with enough latency that it needs a filler phrase to cover the gap — isn’t described as “the system,” full stop. It’s named as a specific rung, explicitly demoted to fallback status once a faster, natively real-time voice stack becomes primary. Nothing about that demotion deletes the older stack; it just stops being rung one. That’s what a single uptime number can’t express: a system can be fully “up” by the SLA while running its second-best rung the whole time, and nobody watching only the dashboard would know which rung answered the phone.

A softer version of the same idea governs how a security fix for the call-answering endpoint got rolled out: instead of flipping enforcement on against an untested assumption, the new check ran in shadow mode — logging every call it would have rejected, rejecting none of them, until there was real data on whether the assumption held. A ladder rung built for a rollout instead of a live outage: fail open and observe, before you’re willing to fail closed.

Presence as a UX primitive, not a feature

The fourth piece doesn’t show up in reliability engineering, because it isn’t a reliability problem. It’s a design bet: that a private-office agent should feel like a person who happens to be busy, not a service that happens to be available.

The mechanism for that is a route I designed and never built, and I want to be exact about that distinction rather than let it read as shipped. The idea: when a landline-style device goes off-hook, instead of picking up on a fixed, machine-regular schedule, the call routes through a handler that rings a random number of times, each ring a randomized length, with randomized gaps between them — two to five rings, each ring 1.2 to 2.8 seconds, gaps of 0.8 to 1.5 seconds — before the agent answers in a slightly different register than it uses on the cell line, more like an intercom being picked up than a phone. The reasoning behind it, in the exact words from the design note: “the randomness is the whole point — it makes him feel like a person who was doing something else.” A fixed ring pattern announces a machine. A slightly different one every time is what a person’s timing actually looks like, because a person isn’t standing by the phone.

That route is UNBUILT. What exists today is two config values — a minimum and maximum ring count — sitting in the live system, unused, waiting on hardware that was never ordered. I’m naming it here because presence-via-unpredictability is a real, reusable UX primitive that nothing in voice-assistant design vocabulary currently names, and I’d rather call a designed mechanism honestly unbuilt than let an idle config value read as a shipped feature.

Where the category boundary actually sits

None of this is a claim that private-office agents are simply “better” voice bots. They’re solving a different problem, which is why borrowing the neighbors’ vocabulary doesn’t work.

A call-center bot’s whole job is task completion inside one conversation: qualify a candidate, answer a support question, book a slot. It doesn’t need a truth hierarchy, because it has no persona continuous across someone’s whole life to protect — a script and a transcript, and if the transcript is wrong you fix the next call. A recruiting-triage agent I run on the same infrastructure is exactly this shape: fast dispatch model, quality reply model, a fixed regression-locked eval bank it has to keep passing. It never needs to know who I’m having dinner with next week, and there’s nothing for a reversibility taxonomy to gate beyond “should this candidate get an outbound call” — one narrow action class, not a sprawling life.

A chat assistant doesn’t have the phone-line stakes at all. A text-only legal-advice persona I run, bounded by the actual rules for who’s allowed to give legal guidance and to whom, never answers a live call as me, never has a text-to-speech engine that reads a bracketed stage direction aloud to a real caller, and never needs a ring-timing primitive, because there’s no ring. Its risk surface is what it says, not what it does in the physical world on my behalf — no phone picks up, no appointment moves, no call gets placed.

The private-office agent sits in the middle of both and inherits the worst of each: the continuous persona and memory a task-completion bot doesn’t carry, plus the live, physical-world stakes a chat assistant never faces. That combination is the actual category boundary — not “does it use a fancier model,” but “does answering wrong land on a real phone call, in real time, as a person other people already trust.” It’s a live-conversation test, not a sync-versus-async one — a chat reply someone acts on immediately still isn’t a call landing on a person who thinks they’re talking to me right now, which is the stake the four primitives above exist to hold.

What I don’t know yet

I’ve built and can point to a working version of three of the four primitives — the truth hierarchy, the reversibility-gated action tiers, and the degradation ladder each exist in running code or a rendering rule I can show. The fourth, presence design, exists only as a design document and two idle config values. I don’t know yet whether randomized ring timing actually reads as “a person” to a real caller who doesn’t know it’s engineered, because the hardware to test it hasn’t been bought — a real gap between a mechanism I believe in and one I’ve measured.

I also don’t know whether these four primitives are the complete list or just the four that happened to surface first, out of incidents I happened to have. Other pieces of this build touch adjacent ground — server-side identity resolution instead of a self-asserted caller ID, a disclosure gate before content reaches the line, a billing path kept separate enough that one subscription failing doesn’t take the whole persona down with it — but whether those add up to named category primitives the way the four above do is UNKNOWN. I’m not going to claim a complete taxonomy off one system.

And I don’t know if any of this survives a second implementation. Every mechanism here was derived from one person’s estate, one persona, one phone line. A design grammar proven once is a hypothesis, not a category, until somebody builds a second private-office agent under this vocabulary and finds out which primitive generalizes and which one is an artifact of how this particular system happened to fail. Until that exists, this stays what it is: one working instance, one unbuilt route, and a name for the gap I kept hitting in the literature.

— The Private Office

← back to writing