Architecture: Local Agent and Audio Pipeline#

Note: This document was generated by an AI agent.

Overview#

One spoken turn is four network round trips — transcribe, generate, synthesise, and play — run by a worker on core 0 while the render loop keeps drawing on core

1. The conversational agent covers what makes a turn start. This document is about what happens between that moment and the first sound, which is almost entirely a question of what each stage is allowed to cost in time, in bytes, and in contiguous heap.

Three constraints shape every decision below, and none of them is negotiable:

• A useful utterance does not fit in RAM. Anything that would hold one entire must not exist.
• An HTTP handler and the render loop share a core. Anything that blocks belongs on the worker, and the worker must never be waited on from a frame.
• The board speaks no TLS and holds no keys. Everything off-device is plain HTTP on the LAN, including the cloud providers, which are reached through a proxy that owns the credentials.

---

The provider boundary#

The streaming provider interface exposes operations to begin a session, start a turn, send audio in chunks, end a turn, poll for progress, and cancel. Polling reports when the first audio is ready, when the reply has ended, and whether it failed. The optional transcript and reply accessors return borrowed pointers, not owned strings, valid from the poll that reports the reply has ended until the next turn starts. Consumers must respect that lifetime.

The main controller selects the local agent when transcription is enabled, otherwise a stub agent. The stub consumes no real speech and produces timed events for UI testing; with audio enabled its first-audio event starts a diagnostic tone, which is what makes a face-and-panels turn testable with no services running at all.

The reply provider is selected independently of capture and synthesis. A settings store persists the choice among Ollama, Gemini, Claude and OpenAI and the model name for each. It holds no API keys. The cloud providers are reached through a proxy running on the LAN machine, which owns the keys and the per-provider request shapes — including which thinking setting each Gemini model accepts, a detail that has no business on a microcontroller.

Why the contract insists on chunks#

The streaming provider contract requires audio to arrive in chunks and never as a complete utterance. That is not a stylistic preference; it is the only shape that fits:

16-bit mono at 16 kHz          = 32,000 bytes per second of speech
largest contiguous block       ≈ 53 KB   (max_alloc, no PSRAM)
                                 53,000 / 32,000 = 1.65 s

One and a half seconds is not a sentence. Recording an utterance and then sending it is therefore impossible on this board at any useful length, whatever the API might prefer — so the audio is streamed while it is still being spoken, and the only thing held in memory is a fixed jitter buffer:

WHISPER_UPLINK_RING_SAMPLES 4000 = 8,000 bytes = 250 ms at 16 kHz

Eight kilobytes, whether the talker speaks for two seconds or two minutes. The capture task fills that ring every 20 ms and the network task drains it as fast as the socket accepts; it is sized well above one chunk so a momentary stall in a socket write costs nothing, and far below the utterance so it never becomes the thing that has to fit. Short writes drop samples rather than blocking, because capture cannot wait — a dropped 20 ms is a far smaller fault than a stalled microphone.

Declaring a length used to cost every turn four seconds#

The upload once declared a fixed Content-Length and padded the remainder with silence, on the reasoning that multipart parsers handle chunked encoding inconsistently. That had two prices, and the second one was invisible:

old:  6 s declared × 16,000 Hz × 2 B  = 192,000 bytes, every turn
new:  a 2 s question                  =  64,000 bytes

A short question cost exactly as much to send as the longest one allowed — and six seconds was also, therefore, a hard cap on how long anyone could speak.

Measured against the server this actually talks to (whisper.cpp on cpp-httplib), a chunked multipart POST answers 200, and so does a WAV header that over-declares its data size, transcribing only the audio that arrived. So the header now declares the watchdog ceiling — the maximum-utterance setting, two minutes — the body carries what was actually said, and the upload ends when the talker does. Silence is what normally ends a turn; the ceiling is a recovery path for a room that never goes quiet.

---

End-to-end turn#

Every service stage is separately gated at compile time. Ollama streams into a bounded reply buffer; a cloud reply is parsed after the proxy answers. Piper starts only after generation returns — sentence splitting overlaps synthesis with already-queued playback, but it does not start speaking the first sentence while the model is still producing the second.

One subtlety worth knowing: the first reply fragment can mark first-audio before any audible PCM exists. The voice-turn state machine uses it to enter Speaking and show the ticker, and the amplitude renderer holds the mouth closed until a real level arrives — which is exactly the right picture for the gap.

---

Where the time goes#

The number that matters is not the turn's total duration but the interval from the last word spoken to the first sound heard, because that is the part the listener experiences as the device thinking.

Two of those six seconds are structural and cannot be recovered by faster services: the 0.9 s silence window is how the device knows the turn ended at all, and the 1.2 s is the shortest synthesis that can produce anything audible. The 2.9 s is the one worth attacking, and the lever is residency rather than speed — llama3.2:1b stays loaded and generates at 40 tok/s, while phi4-mini loads in 12.6 s, is evicted between turns, and manages 16–17 tok/s.

Every deadline is an order of magnitude above its measurement#

Generous on purpose. A deadline that fires early is reported as "the agent failed", which sends you looking at the microphone.

The gaps are deliberate. A timeout that trips on a slow-but-working service is worse than no timeout at all, because the failure it reports — the agent failed — sends you to the microphone, the wake word and the network before the one number that was actually wrong. A dead server still fails, just a few seconds later.

But the deadlines are not independent, and there is one invariant between them. The voice-turn state machine's Thinking state spans everything from the release to the first audio, so every deadline inside it has to fit:

whisper   4 s connect + 60 s reply   =  64 s
ollama    4 s connect + 25 s first   =  29 s
piper     4 s connect + 15 s reply   =  19 s
                                       ----
                                       112 s   <   VOICE_THINK_TIMEOUT_MS 120000

Eight seconds of margin. Raise any service deadline without raising the think timeout and the session gives up before the service does — which presents as a turn that failed for no reason while the server log shows a perfectly good answer arriving a moment later.

Speaking, by contrast, has no state-machine timeout at all: a long reply is legitimate, and the provider reports its own end. A failure event covers the connection dying.

---

Sentence splitting as the unit of TTS latency#

Piper synthesises at roughly 0.4× realtime — a second of speech takes about 0.4 s to produce. Sending the whole reply as one request would therefore make the listener wait for all of it:

a four-sentence reply, 12 s of audio
  whole reply    12 s × 0.4 = 4.8 s before the first sound
  first sentence  3 s × 0.4 = 1.2 s before the first sound

So the speech dispatcher splits the reply on sentence boundaries and sends them one at a time, bounded by a 220-byte sentence limit. The first sentence's share is what appears in the ledger above; the rest is synthesised while earlier audio is still playing.

That overlap has a budget, and it is the playback ring:

VOICE_PLAYBACK_BUFFER_SAMPLES 4000 at 24 kHz = 166.7 ms

The synthesis-playback step pushes into that ring against backpressure, so it returns only once the last sample of a sentence is queued — meaning at most 167 ms of audio remains unplayed when the request for the next sentence begins. Time to first byte, not synthesis rate, is what keeps speech gapless: Piper must start returning PCM for sentence N+1 within 167 ms of finishing N, or the ring underruns and the gap is audible. On a LAN it does; the drop and underrun counters in the audio status report are where it would show if it stopped.

The synthesis streaming timeout is 20 s for the same reason the ring is small: pushing a long sentence legitimately takes that sentence's own duration, because the ring drains at playback speed and nothing can hurry it.

The on-screen ticker is paced from the same split. Each sentence is appended to the ticker before it is spoken, so the line count is known when the duration arrives, and the pace is set to the duration divided by the line count. Deferring the text by the ring's full depth was tried on real hardware and over-corrected — the ring holds 167 ms while a sentence runs for seconds, so the per-line pacing does the real work. A manual sync-hold setting remains as a manual nudge, zero by default.

---

Voice lifecycle and wake gating#

The controller constructs the voice session with a 15 s listen limit, the 120 s think limit derived above, and a 2.5 s error hold. UI cancellation resets the session, stops capture and tone, clears queued playback, and closes the follow-up window. Cancelling from the local agent does not abort an in-flight socket or worker — a known limitation of the current implementation.

Failing audibly, not silently#

The Error state is the exception, not the rule, for a failed turn — most failures never reach it at all. A reply-generation failure (Ollama or the cloud proxy unreachable) completes as an entirely ordinary turn: Thinking → Speaking → Idle, with an apologetic sentence substituted for the answer ("Não consegui obter uma resposta do modelo local agora." for Ollama, or a cloud-specific failure phrase for the cloud path) and spoken exactly like any other reply. This is not new behaviour — it is the existing path every generated reply already takes — but it means the device was never actually silent about that half of a turn failing.

The half that was silent is transcription. A Whisper connection failure used to fail the whole turn outright, straight to the Error state and a generic "agent failed", with nothing spoken — the one place a real failure produced no audible signal at all. It now follows the same shape as the reply-stage case: when the network is up, the worker calls the speech dispatcher directly and synchronously — the same function, the same thread, the same pattern the reply-answering path already uses for its own apology — with a short phrase ("Não consegui te ouvir agora."), and if that succeeds the turn is absorbed into a normal completion exactly like a reply failure is, never entering the Error state. Internal failure counters and the transcription error field still record what actually happened underneath; only the user-facing outcome changes.

The Error state is reserved for the one case where nothing can be spoken at all: the network itself is down, so Piper is equally unreachable. There, the worker falls back to a network-independent diagnostic tone instead, and the sad-face cue is the honest signal that something went wrong. The voice-turn state machine's own two internal timeouts ("held too long", "no reply") still reach the Error state the same way they always did — those aren't Whisper-connectivity failures and have nothing to substitute.

The panel text now outlives the 2.5 s hold, for the case that still uses it. The main controller keeps a separate last-turn-failure field in the panel UI, set once at the moment the Error state is entered — preferring the local agent's specific transcription-error reason over the state machine's own generic one — and cleared only when a fresh turn actually starts, not on a timer. The Voice and Speak tabs read it in preference to the ephemeral error message, so the real reason is still legible if the panel is checked well after the cue itself has reverted to idle, rather than only in the 2.5 s window immediately after.

The gate itself is covered in detail in the conversational agent; what belongs here is which gate a turn runs under:

The name alone becomes an explicit "Vamos conversar" model prompt; text after the name becomes the request. Matching folds tokens for comparison but preserves offsets into the original UTF-8, which is what lets the model be prompted with the question rather than the greeting.

The gate runs on the device, after Whisper, inside the assistant-enabled branch. Sustained nearby utterances therefore reach the configured Whisper host before the phrase is known; non-wake text is then withheld from the model and the synthesiser. This is not local acoustic keyword spotting, and the reason it cannot be is a memory ceiling rather than a design choice. Playback plus a 900 ms cooldown suppresses re-arming; there is no echo cancellation and no supported full-duplex barge-in contract.

An answered wake turn opens or extends a 20 s follow-up window at completion. One counter is worth reading carefully: the answered-turns counter increments when generation is attempted, before success is known, so it is not a count of verified audible answers.

---

External service contracts#

These are the protocols this checkout sends and expects, not a deployment of the services themselves. Hosts default empty; ports are template defaults.

All four are plain HTTP on the LAN. Cloud providers used to be reached directly over HTTPS from the board, which cost three things at once:

That single change is why the memory ledgers elsewhere in these documents close at all — 43 KB of transient heap per request is most of a board whose largest contiguous block is around 53 KB. The proxy also logs each request's prompt size and upstream latency, which is the only place those two numbers can be seen together.

The clients are deliberately minimal#

None of them contains a general HTTP implementation, and each has a specific assumption that a substituted server could break:

• Whisper scans the raw body for the JSON text field.
• Piper expects raw WAV bytes after the headers and examines an initial 128-byte header buffer, so the WAV's chunks and format must fit that parser.
• Ollama scans newline-delimited fragments for the literal "done":true, and can retain a partial reply when text arrived without a terminal marker.
• None decodes chunked responses; the Ollama client explicitly rejects a Transfer-Encoding response header rather than mis-parsing one.

A proxy that changes framing or rewrites WAV metadata will break compatibility in ways that look like a device fault. A keep-alive request asks a server to retain a model but cannot guarantee residency; choose the deadlines for the model actually installed.

---

Natural conversation and recent-turn memory#

Ordinary language uses the selected reply provider and the conversational system prompt. A bounded conversation-history store retains the last four completed user/Boto exchanges in fixed arrays. Before a follow-up request it formats them as bounded <usuário>/<boto> context and places the current transcript in <mensagem_atual>. If space is tight the composer drops complete oldest turns; it never cuts the current message — a truncated question is worse than a missing one, because the model answers it anyway.

A final reply-cleaning guard is the final check before memory and speech. Small completion models can still continue the supplied context as Usuário: … / Boto: …; the cleaner keeps only the final Boto segment, removes any generated next-user turn, and preserves its UTF-8 bytes. Piper therefore receives accented text, while only the separate OLED copy is folded to ASCII for the built-in font.

This makes the acoustic follow-up window semantic as well: after "Meu nome é Iva", a follow-up such as "Qual é meu nome?" reaches the model with that earlier exchange. Memory is deliberately short-lived and local to the active conversation. It is not persisted, never sent to Whisper, and not exposed as a general chat log. A cold wake phrase, a push-to-talk turn, or a diagnostic text request clears it.

Exact allowlisted device phrases still pass through the local phrase matcher before chat generation, so "mostre o relógio" remains deterministic without forcing every ordinary reply through a JSON command schema.

---

Structured device actions#

With device commands enabled, an utterance first passes through the local phrase matcher, an exact Portuguese matcher for common actions. If it does not match, model-routed commands — on by default in the tracked configuration — ask the model through a structured-interpretation call for a single structured JSON object instead of prose; turning that off falls back to natural chat for everything the exact matcher missed.

{"kind": "operation", "operation": "animation.play", "target": "happy", "phrase": ""}

kind is one of chat (ordinary conversation, spoken as phrase), clarify (ambiguous, negated, or compound requests, spoken as a clarifying question — "say one operation at a time"), or operation (exactly one of animation.play, panel.show, face.show, time.get, weather.get, status.get, usage.get, with phrase left empty — and a non-empty phrase even then, since every intent speaks).

The Ollama client asks for this object via an explicit JSON schema (an object schema with all four fields required, kind restricted to its three values) and builds its system prompt from the same catalogue-building routine used for validation, so the model is never offered a target the device would then reject. The cloud path does the same through the proxy, which declares its tools from the catalogue the board sends rather than from a list of its own that would drift.

That catalogue is not free: it costs about 640 extra prompt tokens per cloud turn — measured on this device's real catalogue, where a 65-token prompt becomes roughly 706 — and the usage panel shows the difference immediately.

The validator then independently validates the returned object field by field: an unknown or duplicate field, a missing field, an unlisted operation, a target outside [A-Za-z0-9_-] (lowercased in place before matching), or a chat/clarify kind carrying a non-empty target or an empty phrase all fail closed to no action — never a partial or best-guess execution. The trust boundary stays in C++, where its host tests are.

Lifecycle#

The device dispatcher is the loop-owned arbiter for what happens once a device request is accepted. Its accept step rejects an invalid, obsolete-turn, duplicate, or already-busy request outright; an accepted request then moves through:

accepted -> waiting_for_speech -> started -> completed
                 \                       \-> canceled (turn abandoned, superseded, or failed)
                  \-> rejected (unknown target, disabled style/panel, renderer refusal)

animation.play resolves to Waiting so its spoken acknowledgment finishes before the action-applying step starts the scene — the Start event fires once the acknowledgment finishes playing — reserving it for the scene's own duration before the Complete event fires. The other six resolve immediately to Applied or Completed: they navigate or answer synchronously, with nothing to reserve. A bounded ring of the last eight device action events backs the assistant status endpoint's events array.

Target resolution is pure and host-tested. Dedicated lookup functions match target against slug names derived from the live scene and panel registries — protocol identifiers, not translated display labels, so a renamed scene label never breaks a saved target — and a separate mapping step picks the subset of scenes the Classic renderer can show. time.get, weather.get, status.get and usage.get instead produce a fact-based reply from a live-data snapshot: never a model-generated number, and weather more than double its own refresh interval old is reported as explicitly stale rather than current.

Turn mailbox and diagnostics#

The turn mailbox is the single-operation mailbox between the worker task on core 0 and the render loop on core 1: the worker submits a device request, the loop takes and eventually resolves it, and a result only ever reaches the turn and request id that submitted it. This replaced streaming an action out of the reply text as tokens arrived — a complete object only exists once the schema-constrained JSON has actually finished.

A diagnostic text endpoint submits text through the same code path a spoken turn uses — same-origin, authenticated, and independent of microphone hardware — and an assistant status endpoint reports the turn id, whether it's busy, the last spoken reply, the current action's state and reason, and the event ring. The older forced-action endpoint is preserved for direct dispatch outside a conversational turn, and still returns a not-found error for an unrecognised command name.

To add an operation: extend the operation vocabulary and its name lookup, add its resolution or reply logic to the capability layer, wire the new case into the dispatch step, and add it to the capability catalogue so the model is actually offered it. The model never supplies code, a function name, or anything beyond this fixed, tested vocabulary.

What a cloud turn costs, and who says so#

Money travels from the proxy as integer micro-dollars (1 USD = 1,000,000), because a float drifts once totals are added up and the device only ever displays them. A count of -1 means the provider did not report that figure, which is not the same as zero and is never rendered as one. countingSince travels with the totals for the same reason: the proxy's ledger is not the provider's bill, and a figure shown without its start date invites the reader to compare it against a console total and conclude it is wrong.

---

PCM and lip-sync#

Mic gain is applied before narrowing the sample down to 16 bits, with saturation to avoid overflow. Piper marks each block's peak before compression removes its dynamics, and a small fixed-size queue carries that mark across the playback ring until the speaker reaches the sample position it describes. The arithmetic behind all of it — why compression was needed, what it cost the lip-sync, and how the alignment is exact rather than tuned — is in making the agent audible.

The byte ledger for one turn#

These are payload sizes, not total RAM — rings reserve an extra slot, and DMA, task stacks, networking, display buffers and metadata all consume memory beyond this. Note how little of it is dynamic: the only heap allocation in the list belongs to a bench tool rather than to a turn, which is what makes a two-minute utterance cost no more memory than a two-second one. The contiguous-heap picture those static buffers sit inside is in the audio diagnostics ledger.

---

Failure modes#

---

Configuration reference#

---

See also#

• The conversational agent — what starts a turn, and the wake gate
• Device actions — why a small model's structured output cannot be trusted
• Making the agent audible — the compressor and the lip-sync alignment
• Audio diagnostics — the contiguous-heap ledger