Draft plan ยท 16 Nov 2026

Master voice agent plan

How the voice agents get a memory, how that is proved with tests, and how every future agent is built from one base. Nothing here is built yet.

The research behind it: Voice agent teardown.

1. The problem

Every agent is built from the 12 Nov standard (execution/voice_demo/standard/: base prompt + trade + profile), but the agent has no memory. It runs on Haiku 4.5 with a 15k-character prompt and has to work out from the raw transcript, on every turn, what it has already been told. Nothing checks that it got this right.

The three LDG Joinery calls on 13 Nov show what that costs:

Call What happened Kind of failure
conv_3601m7b2kwfye8f8nq3ks4s5hjh2, 1:04Caller: "Next week if possible. How much would it cost, by the way?" The agent answered the price question, then asked "when are you hoping to get it done?" again.Re-asked a detail it had just been given
conv_7401m7b1ydt3fxz9zqcs6g4s1t8z, 0:58Caller had said "12 Chorley New Road, Bolton". The agent then asked "And which town is that?"Re-asked a detail it had been given
same call, 1:36Caller answered the timing question with "next week would be great". The agent replied "Leon will sort a time with you when he rings back", the line meant for a caller asking for a visit.Treated an answer as a request
same call, 1:36Asked "What's the best number to reach you on?", the wording for a withheld number, though the caller ID was present.Wrong branch
conv_0501m7b2fjqefmy8f2nb8gw92rve, 0:22Caller paused mid-sentence ("...and I need someone-") and the agent cut in.Turn-taking

The first four are one class: the model lost track of the state of the call. The current test pack (21 scripted ElevenLabs text tests) has no case where a caller answers and asks in the same breath, and cannot run our tools, so it could not have caught any of them.

2. Decisions already made (Rachel, 16 Nov)

3. What done means for the first build

  1. The LDG re-ask turn passes 10 runs out of 10 with the record service on, and fails at least once in 10 on today's prompt with the service off. If it never fails on today's prompt, the test is not reproducing the bug and gets fixed first.
  2. Across every rig run: no detail the caller has given is asked for again, and no detail is asked for more than twice.
  3. Every must-pass scenario passes 10 out of 10 (list in section 7).
  4. One handset call with the service on still meets S1 (median under 1.0s to first audio, no turn over 2.5s) and S9 (the lead that reaches the owner matches what the caller said).
  5. With the service switched off mid-test, the call carries on using ElevenLabs' hosted model.
  6. A new agent is still made from a profile, a trade and an accent. No behaviour is edited per company.

4. How it fits together

caller -> Telnyx -> ElevenLabs agent (hearing, turn-taking, voice, runs the tools)
                         |  every turn: the prompt + the conversation so far
                         v
                    brain  (our service on the VPS)
                         |  extraction call   |  speaking call
                         v                    v
                       Claude (Anthropic API, Rachel's key)

if the brain fails: ElevenLabs falls back to its hosted Haiku with the same prompt

ElevenLabs stays in charge of the phone line, speech recognition, turn-taking, the voice and running send_summary and find_postcode. Only the thinking step moves. ElevenLabs calls this a custom LLM: it sends the system prompt and the conversation to an OpenAI-style streaming endpoint and speaks whatever comes back.

Because the hosted fallback runs the plain prompt with no record, the base prompt must stay able to carry a whole call on its own. The brain adds to it; it never replaces it.

5. Step 0: pre-build tests

ElevenLabs' documentation is silent on three things this design depends on. They get tested with a pass-through version of the brain (it forwards to Claude and changes nothing) before any record logic is written.

Question How it is tested If the answer is no
Do send_summary and find_postcode reach the custom endpoint in the tool list, and run when the model calls them?A duplicate test agent with no phone number, in ElevenLabs' text test; both tools called once, pointed at a webhook that sends nothing.Stop. Rebuild the plan around ElevenLabs' own Update state tool and per-turn variables. The rig and prompt work carry over.
What identifies the call in each request?Log one whole request. Look for a conversation ID and the caller's number.Not a blocker: the record can be keyed on a hash of the conversation so far, and a missing record is rebuilt from the transcript.
What does the caller hear when the endpoint is slow or down?Make the pass-through wait 5 seconds, then return an error, then switch it off, with backup_llm_config set to override and hosted Haiku 4.5 in the order.Tune cascade_timeout_seconds (2 to 15s, default 4). If no setting gives a tolerable call, stop as above.
How long does extraction take?Time the extraction call on every caller turn of the five real calls.Decides the wiring in section 6.4.
Do ElevenLabs' own agent tests run through the custom endpoint?Run two of the existing test pack cases against the test agent and check the brain's log.The native suite then tests only the fallback path; the rig and the handset call cover the brain.

Also captured here: one real request saved as a fixture, so the rig imitates ElevenLabs exactly, and how an interrupted agent turn appears in the conversation.

One handset call is needed for the latency numbers. Recommendation: switch the Ambient line (Rachel's original test line) to the pass-through for that one call, using build.py's snapshot and rollback, then put it back.

6. The brain

New service, execution/voice_brain/, FastAPI, deployed to the VPS as a systemd service behind Caddy the same way demo_desk_feed is. One endpoint per agent, with the prospect slug in the URL path so the model never types it (the pattern send_summary already uses). ElevenLabs authenticates with a bearer secret held as an ElevenLabs workspace secret and in .env.

6.1 The flow file

Today the order of questions exists only as prose in base_prompt.md. Code cannot choose the next question from prose, so the order moves into data: standard/flow.json.

build.py renders the numbered steps of "HOW A CALL GOES" in the base prompt from this file, so the prompt the fallback sees and the order the brain enforces come from one place. Trade files keep the job follow-up question and the urgent triggers.

6.2 The call record

One record per call, held in a small SQLite file on the VPS and deleted after 30 days, matching the agents' retention.

The record is always rebuildable from the conversation ElevenLabs sends, so a restart or a lost record costs one slower turn, never a wrong one.

6.3 Extraction

A separate, small Claude call with a fixed output shape. It sees the list of details, the record so far, the agent's last line and what the caller just said, and returns only what changed: details given or corrected, refusals, questions the caller asked, a change of path.

Code, not the model, then does the parts models get wrong (LEARNINGS, 23 Oct):

6.4 Speaking, and the guard

The speaking call gets the base prompt exactly as ElevenLabs sent it, plus a short block built from the record: what is captured, what is still needed in order, and the instruction that anything in the caller's last turn counts as captured.

Extraction and speaking run at the same time, so extraction adds no delay to the first word. The guard sits on the output:

  1. Anything that is not a question (an answer, "Thanks, Rachel.") is passed to ElevenLabs straight away.
  2. The question sentence is held until extraction has finished, for at most about 400 ms.
  3. If the question is about a detail the record now holds, or one already asked twice, it is swapped for the right one from the flow file: the next missing detail's question, or the let-go line followed by the next question. The swap is instant because the wording is already written.
  4. If extraction is late or fails, the question goes out unchanged and the miss is logged.

If step 0 shows extraction finishing in under 300 ms at the 95th percentile, it simply runs first and the hold is dropped. A question the patterns do not recognise is let through and logged; in the rig it is a failure, which keeps the patterns complete.

6.5 The two-ask cap

Counted in code from the conversation, per detail. First ask: the normal question. Second ask: the different-method wording. After that the guard replaces any third ask with the let-go line ("No problem, Leon can confirm that when he rings.") and the detail is flagged in the summary as unsure. A refusal is final on the first no.

6.6 Sending the lead

When the model calls send_summary, the brain fills the name, number, address and timing from the record and keeps the model's one or two sentence summary. A second send_summary in the same call is dropped. ElevenLabs still runs the webhook, so n8n, the texts and the desk card are untouched.

6.7 When things fail

Failure What happens
Extraction errors or times outReply goes out unguarded, record unchanged, logged
Claude errors before the first wordThe brain returns an error; ElevenLabs retries, then falls back to hosted Haiku
Brain or VPS unreachableElevenLabs falls back to hosted Haiku with the same prompt
Repeated failuresTelegram alert to Rachel through the existing bot

Every turn writes one log line: timings (extraction, first word, hold), the record, and whether the guard changed anything. No secrets, and caller details are deleted with the record. The log gives a real re-ask rate from live calls, which nobody in this market publishes.

6.8 Model

The model is a setting. The first build uses Haiku 4.5, the model the agents run today, so before and after are compared like for like. The rig then runs the same scenarios on Haiku 5.5, and the agents move only if it passes at least as many within the latency budget. Sonnet stays out: LEARNINGS records it as about a second slower with no obedience gain.

7. The test rig

execution/voice_brain/rig/. It plays ElevenLabs' part: sends the brain the same request shape captured in step 0, receives the streamed reply and tool calls, and answers tool calls with mocks. Nothing real is sent, no number is rung, no card lands on a desk.

Two kinds of scenario:

Checks are code wherever code can decide:

A judging model is used only where code cannot decide ("did it answer what the caller asked", "did it avoid confirming a claim that is not in the profile"), and those results are reported separately with the judge's reason.

Each scenario runs 10 times. Must-pass scenarios need 10 out of 10: no re-ask of a given detail, the two-ask cap, lead sent once and correct, never prices or books, safety first, honest when asked if it is AI, sales calls brushed off. The rest are reported with their pass count. Every run can be repeated with the brain in pass-through mode. Run that way on today's prompt, before the changes in section 8, it gives the before figure. Spend is capped with the existing SpendGuard in execution/llm_spend.py.

build.py --push for an agent that uses the brain refuses to push unless the rig has passed on the current base and flow file.

What the rig cannot see: interruptions, pauses and anything else that happens in audio. That stays with the existing ElevenLabs test pack, run against the brain-enabled agent, and the handset call. score_call.py gets the same ask counter in place of its current rough one.

8. Changes to the standard

9. Call history and the failure catalogue

Separate plan, started when Roman's key arrives; it can begin on Rachel's five calls.

10. After the first build (each gets its own spec)

  1. Emergency live transfer. After the safety line, on the trade's urgent triggers: put the caller on hold, ring the owner, give a spoken briefing, connect on a spoken yes, fall back to taking a message and an alert text if it reaches voicemail. One attempt per call. It dials out from the shared Telnyx account, so Roman agrees how it works before anything rings.
  2. Knowledge base. Company facts built from the website and Google listing before the call, treated as facts only, with anything that reads like an instruction stripped. Unknown questions still get "Leon will confirm that when he rings."
  3. Calendar sync and booking, per client. A switch in the profile. Step one texts a booking link; step two reads and writes one Google or Outlook calendar. The rule in CLAUDE.md that demos never book is reworded when this ships.
  4. Later, where simple enough for a demo: a link or photo request texted mid-call, postcode lookup run by the brain itself, a returning-caller brief, paths for complaints and business callers, a test mode that sends nothing.

11. Risks and open points

12. Order of work

  1. Plan 1: step 0 (pass-through brain, the five tests) and the rig skeleton with the four LDG next-reply tests, run in pass-through mode on today's prompt to record the before figure.
  2. Plan 2: the brain (flow file, record, extraction, guard, two-ask cap, lead fill, logging, fallback), the full scenario set, the changes to the standard, then LDG, Ambient, AJV and BuildSpark rebuilt on it.
  3. Plan 3: call history and the failure catalogue, when Roman's key arrives.