Draft plan ยท 16 Nov 2026
How the voice agents get a memory, how that is proved with tests, and how every future agent is built from one base. Nothing here is built yet.
Every agent is built from the 12 Nov standard (execution/voice_demo/standard/: base prompt + trade + profile), but the agent has no memory. It runs on Haiku 4.5 with a 15k-character prompt and has to work out from the raw transcript, on every turn, what it has already been told. Nothing checks that it got this right.
The three LDG Joinery calls on 13 Nov show what that costs:
| Call | What happened | Kind of failure |
|---|---|---|
conv_3601m7b2kwfye8f8nq3ks4s5hjh2, 1:04 | Caller: "Next week if possible. How much would it cost, by the way?" The agent answered the price question, then asked "when are you hoping to get it done?" again. | Re-asked a detail it had just been given |
conv_7401m7b1ydt3fxz9zqcs6g4s1t8z, 0:58 | Caller had said "12 Chorley New Road, Bolton". The agent then asked "And which town is that?" | Re-asked a detail it had been given |
| same call, 1:36 | Caller answered the timing question with "next week would be great". The agent replied "Leon will sort a time with you when he rings back", the line meant for a caller asking for a visit. | Treated an answer as a request |
| same call, 1:36 | Asked "What's the best number to reach you on?", the wording for a withheld number, though the caller ID was present. | Wrong branch |
conv_0501m7b2fjqefmy8f2nb8gw92rve, 0:22 | Caller paused mid-sentence ("...and I need someone-") and the agent cut in. | Turn-taking |
The first four are one class: the model lost track of the state of the call. The current test pack (21 scripted ElevenLabs text tests) has no case where a caller answers and asks in the same breath, and cannot run our tools, so it could not have caught any of them.
caller -> Telnyx -> ElevenLabs agent (hearing, turn-taking, voice, runs the tools)
| every turn: the prompt + the conversation so far
v
brain (our service on the VPS)
| extraction call | speaking call
v v
Claude (Anthropic API, Rachel's key)
if the brain fails: ElevenLabs falls back to its hosted Haiku with the same prompt
ElevenLabs stays in charge of the phone line, speech recognition, turn-taking, the voice and running send_summary and find_postcode. Only the thinking step moves. ElevenLabs calls this a custom LLM: it sends the system prompt and the conversation to an OpenAI-style streaming endpoint and speaks whatever comes back.
Because the hosted fallback runs the plain prompt with no record, the base prompt must stay able to carry a whole call on its own. The brain adds to it; it never replaces it.
ElevenLabs' documentation is silent on three things this design depends on. They get tested with a pass-through version of the brain (it forwards to Claude and changes nothing) before any record logic is written.
| Question | How it is tested | If the answer is no |
|---|---|---|
Do send_summary and find_postcode reach the custom endpoint in the tool list, and run when the model calls them? | A duplicate test agent with no phone number, in ElevenLabs' text test; both tools called once, pointed at a webhook that sends nothing. | Stop. Rebuild the plan around ElevenLabs' own Update state tool and per-turn variables. The rig and prompt work carry over. |
| What identifies the call in each request? | Log one whole request. Look for a conversation ID and the caller's number. | Not a blocker: the record can be keyed on a hash of the conversation so far, and a missing record is rebuilt from the transcript. |
| What does the caller hear when the endpoint is slow or down? | Make the pass-through wait 5 seconds, then return an error, then switch it off, with backup_llm_config set to override and hosted Haiku 4.5 in the order. | Tune cascade_timeout_seconds (2 to 15s, default 4). If no setting gives a tolerable call, stop as above. |
| How long does extraction take? | Time the extraction call on every caller turn of the five real calls. | Decides the wiring in section 6.4. |
| Do ElevenLabs' own agent tests run through the custom endpoint? | Run two of the existing test pack cases against the test agent and check the brain's log. | The native suite then tests only the fallback path; the rig and the handset call cover the brain. |
Also captured here: one real request saved as a fixture, so the rig imitates ElevenLabs exactly, and how an interrupted agent turn appears in the conversation.
One handset call is needed for the latency numbers. Recommendation: switch the Ambient line (Rachel's original test line) to the pass-through for that one call, using build.py's snapshot and rollback, then put it back.
New service, execution/voice_brain/, FastAPI, deployed to the VPS as a systemd service behind Caddy the same way demo_desk_feed is. One endpoint per agent, with the prospect slug in the URL path so the model never types it (the pattern send_summary already uses). ElevenLabs authenticates with a bearer secret held as an ElevenLabs workspace secret and in .env.
Today the order of questions exists only as prose in base_prompt.md. Code cannot choose the next question from prose, so the order moves into data: standard/flow.json.
build.py renders the numbered steps of "HOW A CALL GOES" in the base prompt from this file, so the prompt the fallback sees and the order the brain enforces come from one place. Trade files keep the job follow-up question and the urgent triggers.
One record per call, held in a small SQLite file on the VPS and deleted after 30 days, matching the agents' retention.
The record is always rebuildable from the conversation ElevenLabs sends, so a restart or a lost record costs one slower turn, never a wrong one.
A separate, small Claude call with a fixed output shape. It sees the list of details, the record so far, the agent's last line and what the caller just said, and returns only what changed: details given or corrected, refusals, questions the caller asked, a change of path.
Code, not the model, then does the parts models get wrong (LEARNINGS, 23 Oct):
find_postcode result is read from the tool's reply, not from the model's wording.The speaking call gets the base prompt exactly as ElevenLabs sent it, plus a short block built from the record: what is captured, what is still needed in order, and the instruction that anything in the caller's last turn counts as captured.
Extraction and speaking run at the same time, so extraction adds no delay to the first word. The guard sits on the output:
If step 0 shows extraction finishing in under 300 ms at the 95th percentile, it simply runs first and the hold is dropped. A question the patterns do not recognise is let through and logged; in the rig it is a failure, which keeps the patterns complete.
Counted in code from the conversation, per detail. First ask: the normal question. Second ask: the different-method wording. After that the guard replaces any third ask with the let-go line ("No problem, Leon can confirm that when he rings.") and the detail is flagged in the summary as unsure. A refusal is final on the first no.
When the model calls send_summary, the brain fills the name, number, address and timing from the record and keeps the model's one or two sentence summary. A second send_summary in the same call is dropped. ElevenLabs still runs the webhook, so n8n, the texts and the desk card are untouched.
| Failure | What happens |
|---|---|
| Extraction errors or times out | Reply goes out unguarded, record unchanged, logged |
| Claude errors before the first word | The brain returns an error; ElevenLabs retries, then falls back to hosted Haiku |
| Brain or VPS unreachable | ElevenLabs falls back to hosted Haiku with the same prompt |
| Repeated failures | Telegram alert to Rachel through the existing bot |
Every turn writes one log line: timings (extraction, first word, hold), the record, and whether the guard changed anything. No secrets, and caller details are deleted with the record. The log gives a real re-ask rate from live calls, which nobody in this market publishes.
The model is a setting. The first build uses Haiku 4.5, the model the agents run today, so before and after are compared like for like. The rig then runs the same scenarios on Haiku 5.5, and the agents move only if it passes at least as many within the latency budget. Sonnet stays out: LEARNINGS records it as about a second slower with no obedience gain.
execution/voice_brain/rig/. It plays ElevenLabs' part: sends the brain the same request shape captured in step 0, receives the streamed reply and tool calls, and answers tool calls with mocks. Nothing real is sent, no number is rung, no card lands on a desk.
Two kinds of scenario:
Checks are code wherever code can decide:
send_summary called exactly once, and its contents equal the fact sheet;test_pack.py;A judging model is used only where code cannot decide ("did it answer what the caller asked", "did it avoid confirming a claim that is not in the profile"), and those results are reported separately with the judge's reason.
Each scenario runs 10 times. Must-pass scenarios need 10 out of 10: no re-ask of a given detail, the two-ask cap, lead sent once and correct, never prices or books, safety first, honest when asked if it is AI, sales calls brushed off. The rest are reported with their pass count. Every run can be repeated with the brain in pass-through mode. Run that way on today's prompt, before the changes in section 8, it gives the before figure. Spend is capped with the existing SpendGuard in execution/llm_spend.py.
build.py --push for an agent that uses the brain refuses to push unless the rig has passed on the current base and flow file.
What the rig cannot see: interruptions, pauses and anything else that happens in audio. That stays with the existing ElevenLabs test pack, run against the brain-enabled agent, and the handset call. score_call.py gets the same ask counter in place of its current rough one.
base_prompt.md: gains one rule for the fallback path (a caller's turn can hold an answer and a question; take the answer first), and its numbered steps are rendered from flow.json.brain switch per agent. On: build.py sets the custom endpoint and backup_llm_config to override. Off: hosted Haiku as today. Turning it off is the rollback.standard/accents/<name>.json holds the voice, the local place names for the speech recogniser and any regional phrasing. A profile names an accent instead of repeating a voice ID and a town list.build.py read-back: adds the endpoint, the fallback setting and the timeout to the settings it asserts after every push.Separate plan, started when Roman's key arrives; it can begin on Rachel's five calls.
corpus/ folder. Roman's key lives in .env as ROMAN_ELEVENLABS_READ_KEY.CLAUDE.md that demos never book is reworded when this ships.patient setting while an address is given, but the July tests found 4 to 15 second gaps with it, so it is tried only on a handset.