Architecture: Device Actions#

Note: This document was generated by an AI agent.

Overview#

Boto could always talk. Getting it to also do something — play an animation, open a panel, tell you the real time instead of a guess — is a harder problem than it sounds, because the thing deciding what to do is a small language model running on a home LAN, and a small model is not a reliable narrator of its own intentions.

The design that came out of that is not "trust the model more". It is to treat the model's output as an untrusted message from a remote peer, which is what it literally is, and to put a parser and a validator in front of it — the same thing one would do for any other network input, and for the same reasons.

Two lanes, one gate. The fast lane skips the model, not the validation — it reaches the action already carrying a target the device chose itself.

---

Structured output is not a free request#

The earlier design asked the model for a tiny envelope around every reply — a command name and a sentence to say, {"cmd": "sad", "phrase": "Sinto muito."} — and it mostly worked. "Mostly" hid a cost that only became visible when the same model was asked something with nothing to do with an action:

"Quanto é 17 vezes 23?"

Asked as plain conversation, llama3.2:1b answers 17 × 23 = 391, correctly. Asked to wrap that answer in the envelope, the same model forgot the arithmetic and replied "O Boto estou feliz!".

That is the finding the whole design rests on: asking a small model for structure spends capacity it was using for the answer. The formatting is not free, and the old envelope was charging for it on every single reply, whether anyone wanted an action or was simply talking.

The measured comparison, over the same six questions, is recorded in the conversational agent: llama3.2:1b scored about 3 of 6 on command selection, phi4-mini 5 of 6, and a separate classification call — asking a second time, just for the category — scored 1 of 6. Three attempts to fix it by prompting made it worse: rewording collapsed everything to happy, and temperature 0 made it consistently wrong.

Two conclusions follow, and they pull in opposite directions:

1. The model is the unreliable part, not the plumbing. No amount of parsing discipline makes a 1B model choose correctly.
2. Therefore the plumbing must be the part that cannot fail. A model that is wrong 3 times in 6 is acceptable only if being wrong produces a rejection rather than an action.

So the cost is paid deliberately rather than universally. Exact phrases never reach the model at all, and everything the model does return is assumed hostile until proved otherwise.

---

The fast lane: phrases that never need a model#

The local phrase matcher matches a handful of complete Portuguese phrases on the device — naming an emotion, asking for a panel by name — before any network request exists. It normalises first: folds UTF-8 to ASCII, lowercases, collapses runs of whitespace, strips trailing punctuation, and removes a leading "por favor". So "Por favor, fique triste!" and "fique triste" are the same phrase.

Three properties keep it from becoming a second, worse model:

• Exact complete phrases only. Mentioning sadness is not a sad-face command. Anything that is not a whole recognised phrase falls through.
• Compound, negated and overlong requests are clarified, not guessed. Two operations in one sentence, or a negation, returns a clarifying question rather than picking one. An utterance of 512 bytes or more is clarified outright without being examined.
• An unknown animation name falls through to the model, so the rejection that follows is honest rather than silent.

The reward is that the most common commands cost nothing: no round trip, no tokens, no opportunity for the model to be creative. This lane is why device commands can be on while model-routed commands are off and the device still obeys direct instructions.

---

The slow lane: constrain the shape, then distrust it anyway#

When the exact matcher does not match, the request goes to the model — but not as a polite request for JSON. Ollama is given an explicit JSON schema: an object with all four fields required and kind restricted to its three values. The cloud path does the same through the proxy, which declares its tools from the catalogue the board sends rather than from a list of its own that would drift.

{"kind": "operation", "operation": "animation.play", "target": "happy", "phrase": ""}

Constraining the shape at the API level is what turned "usually valid" into "always well-formed or explicitly rejected". But well-formed is not the same as true, and the validator that follows assumes nothing.

What the validator actually checks#

The validator is pure, bounded, and shared with the host tests. It uses a small strict JSON reader rather than searching for substrings — strstr() on a model's output is how a mention of an operation becomes an execution of it. Every one of these rejects outright:

The ninth row has a story. Every intent speaks, operations included, because a tool call that acts without saying anything turns a conversational turn into a silent gesture — the measured failure where "oi tudo bem" produced a wave and no words. Requiring a non-empty phrase is what makes that unrepresentable.

The target is also lowercased in place during that charset scan, because the capability-matching layer matches canonical names in lowercase only and the JSON schema does not constrain case. One pass does both jobs.

The catalogue the model is offered is built from what exists#

The catalogue-building routine composes the string sent with every request, and it is built from the live registries rather than a list maintained by hand:

• animation.play targets come from the renderer's scene registry. Disabled scenes and empty user slots are never offered, and with the Classic renderer selected, scenes it cannot express are filtered out.
• panel.show targets come from the panel screen list, filtered by which are enabled, and a compile-time check keeps that list paired with the panel count so the capability names and the screens cannot drift apart silently.
• face.show, time.get, weather.get, status.get and usage.get take no target.

So the model is never offered a target the device would then reject — and because the validator resolves against the same registries, a target that was valid when the catalogue was built and disabled a moment later still fails closed.

Names are protocol identifiers, not display labels. A user-uploaded scene is addressed as user_3 rather than by its uploaded name, because uploaded names collide, contain non-ASCII, and get renamed; the slot id does not. There is one small concession to how models actually speak — think, sleep and greet are aliased to thinking, sleepy and greeting before matching.

Proven negatively, on the host#

The trust boundary is four pure headers with no Arduino API, so the whole of it runs in the native suite:

(Assertions as the test runner counts them at run time, not source lines — several cases sweep a whole registry.)

Most of the intent suite is deliberately negative — truncated JSON, trailing garbage, a missing field, a duplicated field, an unknown field, an unknown kind, an unknown operation, an oversized field, an oversized input, a lone surrogate, a chat carrying an operation, an operation missing its phrase, a target whose presence contradicts its operation, a charset violation. The behaviour that matters is what the boundary refuses, so that is what is tested.

---

One request, one lifecycle#

Deciding an action is allowed is only half the problem. The other half is that exactly one thing happens, at the right moment, and the device always knows honestly what state it is in.

Every request is exactly one of these at any moment — never zero, never two.

The device dispatcher is the loop-owned arbiter. Its accept step refuses four distinct conditions before anything else happens, each with its own recorded reason:

A rejection is not silent: it carries a spoken reply — "Não posso executar esse pedido agora." — and is recorded in the event ring like any other outcome.

Why an animation waits#

An animation is the case that needed real bookkeeping. Boto says "Certo, vou mostrar a animação" before the scene appears — and the talking mouth owns the screen while it speaks, so a scene started immediately would play behind its own acknowledgment.

So animation.play resolves to Waiting, and the periodic update step promotes it to Start only once the acknowledgment has finished playing. The dispatcher then records the moment and the scene's own duration, and the request stays Started — reserving the screen — until that duration elapses, at which point Complete fires.

Everything else resolves immediately to Applied or Completed: a panel, a return to the face, or a spoken fact has nothing to watch play out and nothing to reserve.

The periodic update step also re-checks the turn on every call, so an action whose conversation was abandoned is cancelled rather than arriving late into a different one. That single check is why a stale request cannot sneak in behind a new turn.

The mailbox between the two cores#

The worker runs on core 0 and the dispatcher on core 1, so the request crosses a boundary. The turn mailbox is a single-operation mailbox carrying plain messages, no borrowed strings — the request and result are fixed structs with inline character arrays. A result only ever reaches the turn and request id that submitted it.

This replaced an earlier approach that streamed an action out of the reply text as tokens arrived. The reason it had to go is exact: a complete object only exists once the schema-constrained JSON has finished, and anything parsed from a prefix is a guess about what the model was about to say.

---

Answers that are actually true#

time.get, weather.get, status.get and usage.get never go near the model's imagination. They are answered from a live-data snapshot of the same facts — the same values the information panels draw.

The whole point of that function is what it does when it does not know:

A count of -1 from the reply proxy means not reported, which is deliberately not the same as zero and is never spoken as one. This is the project's never-invent-data rule applied where it is easiest to break: a sentence sounds equally confident either way.

The model never gets a vote on what the answer is — only on whether the question was asked at all.

---

Adding an operation#

1. Extend the operation vocabulary and its name lookup.
2. Add its resolution or reply logic to the capability layer, or to the fact-reply step if it answers from a snapshot.
3. Wire the case into the request-dispatch step.
4. Add it to the capability catalogue, or the model is never offered it.
5. Add the negative tests first — what should this operation refuse?

The model never supplies code, a function name, or anything outside this fixed, tested vocabulary. Step 4 is the one that is easy to forget and produces the most confusing symptom: an operation that works perfectly through the diagnostic text endpoint and that the model never once chooses.

---

Diagnosing it#

The diagnostic text endpoint submits text through the same code path a spoken turn uses — same-origin, authenticated, and independent of the microphone. That is what made this whole subsystem testable before the audio hardware was trusted.

The assistant status endpoint reports the turn id, whether it's busy, the last spoken reply, the current action with its state and reason, and a ring of the last eight device action events. The voice status endpoint additionally reports the capability catalogue and its length, because an empty catalogue looks exactly like a model that chose not to act.

The older forced-action endpoint remains for direct dispatch outside a conversational turn, and still returns a not-found error for an unrecognised command name.

Verified on the physical device#

The firmware was flashed and driven through real requests over the device's own web API — no simulated hardware:

• Asking for the time returned the real clock, matching a phone held beside it.
• Asking for an animation ran the full lifecycle and the scene appeared.
• Asking for a panel navigated the display to it.
• Asking for something the device does not have rejected cleanly.

The last one matters as much as the first three: a system that is honest about what it cannot do is what makes the rest of it trustworthy.

---

Failure modes#

---

Configuration reference#

With both off, Boto is conversational only. With only the first on, it obeys exact instructions and chats about everything else — the right setting for a 1B model, given the measurements above.

---

See also#

• The conversational agent — the wake word, and the model comparison this page rests on
• Local agent pipeline — where the catalogue and the validator sit in a turn
• Runtime — the render loop the dispatcher is owned by