The runtime#

Note: This document was generated by an AI agent.

Everything on this device is paced by one number: a frame every 33 ms, forever, whether anything changed or not. This page is about what fits inside that, what had to be moved out of it, and how three modes and one touch pad turn into the behaviour you actually see.

---

Why a fixed interval, and never a blocking one#

The loop is deliberately shaped like this: each cycle it checks the time, services OTA and the web server unconditionally, then returns immediately without doing any animation work unless the frame interval has elapsed — only once it has does it update the last-frame timestamp and run the frame update.

Three properties are worth drawing out, because each is a decision.

It returns rather than delays. There is no delay() anywhere in the frame path. A delay(33 - elapsed) would be simpler and would also hand the CPU to nothing — on a cooperative loop, time spent sleeping is time the web server and OTA are not serviced. Returning early means the loop spins, and spinning is the point: the cheap work runs thousands of times a second while the expensive work runs thirty.

OTA and the web server sit before the gate. This looks like a mistake and is the opposite. Gated behind the 33 ms limiter they would run only 30 times a second, and the OTA transfer tool times out waiting for acknowledgements during a transfer. TCP wants servicing at loop rate. They are also deliberately outside the main per-frame update routine, whose animation-only test mode would otherwise skip them — bring-up mode would silently lose the ability to reflash itself.

A missed deadline degrades rather than drifts. The last-frame timestamp is reset to the current time, not advanced by a fixed step. If a frame overruns, the next one starts late and the animation is briefly slower; it never tries to catch up by running two frames back to back, which on a one-bit panel would read as a visible jump.

---

Where a frame goes#

That is the whole budget, and the split is brutal: three quarters of every frame is already spent before any of this firmware's own work begins.

The draw cost inside that 8.3 ms is not asserted here because it depends on what is being drawn — a Classic expression and a 2.5D projection are not comparable. The firmware can measure it for you: enable a timing-log setting and the 3D face renderer reports peak draw and peak transfer microseconds over serial every two seconds, separated, so you can see which half is costing you.

This is the budget every other ledger in this documentation spends from. The 2.5D rig takes its share of the 8.3 ms; the audio runtime explains why it takes none of it at all.

Two rates, not one#

The 100 ms panel interval is not a performance compromise — it is a deliberate refusal to spend the bus on something that has not changed.

---

Boot and frame scheduling#

The firmware's setup routine starts serial at 115200, initializes the display, seeds randomness, then brings up touch input, live data, OTA, the web server, and the application controller in turn. Display allocation/init failure prints an error and stops in a delay loop.

The application controller's startup loads settings and scene order before choosing the first enabled tab, initializes the face and persisted style, optionally initializes bitmap test playback, brings up audio, and starts the selected agent when voice is enabled. A configuration flag selects the local speech-driven agent; otherwise it falls back to a stub, text-only agent.

In bitmap test mode, the frame update returns before touching live data. Web and OTA are still called by the sketch, but network-state polling and AP DNS service are skipped; do not assume this isolated mode has normal network bring-up.

---

One pad, two gestures#

Everything the user can do is built from a single digital pin that reads high while touched. One module owns the pin; another turns its raw signal into gesture events, and that gesture logic is Arduino-free so the host tests can drive it through any press pattern without hardware.

Two design decisions are recorded in that logic and worth repeating:

A long press fires the instant it crosses the threshold, not on release. The gesture feels immediate under the finger — you feel the panel open while still holding. A tap, by contrast, is only reported on release, and only if the long press did not already fire.

Long press was chosen over double tap, and the reason is latency: a double tap forces every single tap to wait out the second-tap window before it can act. Every interaction on the device would have paid for a gesture used rarely.

press ──┬── released before 600 ms ──> GESTURE_TAP   (on release)
        └── still held at 600 ms ────> GESTURE_LONG  (immediately)

A 30 ms debounce window filters contact noise. Note that input is sampled once per frame, about every 33 ms, so a tap shorter than that can be missed entirely — a real limit of polling rather than something the debounce introduces.

---

Three modes#

The current mode, the mode to resume to, panel selection, voice state, follow-up timing and action dispatch all belong to the application controller. The face engine owns animation state; the screens module receives the panel selection and a live-data snapshot and owns neither.

Three rules complicate the diagram, and each exists because the simple version felt wrong in the hand:

• Waking from ambient consumes the gesture. The touch that wakes the device is held through release and never also navigates. Without this, reaching over to wake it would change the expression at the same time.
• Ambient returns to where you were, not to a default. The resume state carries the previous mode and selection. A wake-triggered voice turn restores it too, so being in Panels when you spoke does not dump you on a full-screen face.
• A gesture during a voice turn abandons the turn first, then routes normally. Physical input outranks anything the assistant is doing.

Voice and Speak reinterpret the hold while voice is enabled: a long press starts capture rather than leaving the panel, and releasing ends the utterance. A tap on an active voice tab abandons the turn and advances. Setup behaves as an ordinary tab until a long press focuses its rows, after which Exit is the way out.

The ambient idle timeout (60,000 ms) and panel refresh interval (100 ms) are compile-time constants here. Their similarly-named persisted settings exist but are not yet read by this controller — a gap worth knowing about before trying to change either from the web UI.

---

Task ownership#

The stack sizes are measured, not guessed: the weather worker's source comment records a measured peak of about 2.9 KB with the HTTP client library plus the buffered body, and the speech worker's stack is set to 12,288 bytes for the same reason. Between them the four tasks reserve 28 KB of the ~220 KB heap before any of them does any work — a line item worth remembering when the recorder refuses an allocation.

Core 0 also runs system/network work; it is not an otherwise empty processor. The live-data module polls link/time/battery every 500 ms during normal frames and services AP DNS each call. Its weather worker checks requests every 200 ms. Capture chunks are 320 samples/20 ms; playback chunks are 240 samples/10 ms at template sample rates.

Loop-owned state may be changed directly by web handlers. Workers must hand animation requests back through the controller's action-intake path; they must not select a face scene directly. Audio/agent code uses volatile flags and fixed buffers rather than a comprehensive lock protocol. The ticker is appended by a worker and advanced by the loop. These are existing ownership assumptions, not a general thread-safe API for arbitrary concurrent producers.

---

Cues, actions, and speech rendering#

A character cue carries Listening, Thinking, Speaking, Success, and Error independently of the provider. Classic maps cues to existing expression poses; 3D maps them to collection scenes. Listening/thinking/speaking are held until cleared; success/error expire. The 3D collection saves the interrupted scene and phase while a cue is active and blends back afterward.

For full-screen speech, the frame draw routine renders the speaking face as soon as the session is Speaking or playback is queued. Its mouth follows the measured playback level against a reference value; zero level keeps it shut while generation/synthesis has not yet produced sound. Speak renders the same face at scale 0.70 within panel chrome. The regular face and Speak refresh at 33 ms; other panels use 100 ms by default.

A model action is retained until the voice turn and existing cue finish. The dispatcher then selects the mapped 3D scene and reserves its playback duration before allowing the follow-up Listening cue to take over. This describes the current implementation, even where older source comments describe selection during the turn. Classic ignores 3D scene selection. The diagnostic counter increments when dispatch runs, not when a visible frame is confirmed.

---

The I2S audio boundary#

The audio driver is the only I2S owner. One capture consumer receives signed 16-bit PCM through the agent's audio-send path. A preemptable wake listener yields to manual capture, monitor, or recording. Other active consumers are refused.

The wake-reconciliation logic arms when flags/settings allow it, audio is ready, station Wi-Fi is connected, diagnostics are inactive, and cooldown has passed. It is not restricted to Face mode. A currently feeding wake turn retains capture until VAD ends the utterance, even though the voice state is busy. Playback extends the 900 ms cooldown to reduce self-triggering; there is no acoustic echo cancellation.

See local agent implementation, I2S runtime, and audio diagnostics for transport and device details.

---

When the frame slips#

Everything here is visible rather than logged, which makes the symptom the primary diagnostic.

Runtime invariants#

Timing reference#

---

Practical extension guide#

1. Add a new ambient scene to the ambient scheduler, define its data availability, add rendering in the ambient info drawing routine if needed, and cover scheduling with a host test.
2. Add a panel through the panel identifier enum, setting metadata/defaults, tab labels, and drawing dispatch. Preserve the contiguous panel-setting mapping.
3. Add a provider through the agent interface; preserve chunked I/O and event semantics. Integrate the provider-selection branch and any local-agent-specific ticker, wake, and diagnostics paths explicitly.
4. When adding a turn origin, update both gesture/start logic and the voice-update/wake-reconciliation eligibility checks. Otherwise the next frame can immediately cancel the turn it just started.