Architecture: One Core, No FPU#

Note: This document was generated by an AI agent.

Overview#

Everything in the three documents before this one was measured on a classic ESP32: two cores at 240 MHz, a hardware floating-point unit, and two I2S controllers. The firmware now also builds for a Seeed XIAO ESP32-C6, and that board takes away all three at once.

The missing core is the obvious one and was the least of the problem. The two that cost real work were the missing FPU — this instruction set has no float extension, so every floating-point operation becomes an out-of-line call into a software library — and the single I2S controller, which forces microphone and speaker to share one clock in full duplex. The duplex design is in the I2S runtime.

This document is about what happened when the face and the voice had to run at the same time on that board, because the answer was not what the earlier documents predict, and it overturns one of their load-bearing claims.

Two scarcities, not one#

The first symptoms looked like one problem with one cause. The animation lagged and the audio broke into short bursts that sounded like noise. Both got worse together, both got better together, and both responded to task priorities — so the obvious reading was that one core could not do two jobs, and the fix was to decide which job mattered more.

That reading is wrong, and following it wasted several rounds. Raising the audio tasks above the network stack fixed the noise and froze the face. Raising the task that fills the playback buffer fixed the gaps and starved the renderer. Equalising them let both run and left both mediocre. Every change moved the fault without removing it, which is the signature of a misdiagnosis rather than of a hard trade-off.

The two halves were starved of different resources:

Neither is a scheduling problem. Priorities decide who runs first; they cannot make a transfer shorter or an instruction cheaper. Once the two were separated the fixes were independent, and neither cost the other anything.

Where the frame went: the bus#

Hardware derives the cost of pushing the panel: 1,024 bytes of framebuffer become 9,864 bits with I2C framing, which at 400 kHz is 24.7 ms. Measured on the C6 the transfer took 23.4 ms, consistent with that derivation and the slightly different chunking this platform's I2C library uses.

Against a 33 ms budget that leaves about 8 ms for everything else. On the classic board the renderer owned a core, so 8 ms of slack next to a blocking transfer was comfortable — the transfer yields the processor while the hardware clocks bits out, so the other core kept drawing. On one core there is no other core, and the 3D renderer needed more than 8 ms.

Both documents call that transfer cost "fixed, and not reducible without a faster bus." That was true of the code as written and is not true of the problem. There are two ways to spend less time on a bus: raise the clock, or send fewer bytes.

Sending only what changed#

The display library always writes the whole buffer; it has no partial-update entry point. But between two frames of an animated face, most of the panel is identical. The top and bottom rows are usually blank, and while the character speaks the mouth is often the only thing moving.

The flush now keeps a copy of what the panel is believed to hold, compares it against what was just drawn, and sends only the horizontal bands that differ — each narrowed to the columns that actually changed. The controller is already in horizontal addressing mode, so a band-and-column window addresses a rectangle, which is the same mechanism the library's own full-frame push relies on.

for each band of the screen:
    first, last = first and last column that differs from what is displayed
    if none:  skip this band entirely
    set the band window
    set the column window
    stream only those bytes
remember what was just sent

Per band rather than one window over the whole screen: the eyes and the mouth are vertically separated, and a single bounding box around both would re-send the gap between them. The worst case — every band changed, full width — is about 4% more than a full push, because each band carries its own addressing command. That is the right trade, because the worst case was already the expensive frame and the common case is now nearly free.

Measured, idle, with the 3D face: 103 bytes a frame against 1,042. A frame in which nothing moved sends nothing at all.

The clock the library puts back#

The first measurement of the partial flush said 237 bytes took 23.1 ms. Sending a quarter of the data had bought nothing, which is not a plausible result for a bus — 237 bytes at 400 kHz should be about 5 ms.

The display library manages the bus rate itself. It raises the clock around its own transfers and drops it afterwards, so an unrelated device sharing the bus is not left running fast. Both rates are constructor arguments, and their defaults are 400 kHz during and 100 kHz after. Nothing in this project passed them, so initialisation ended by setting the bus to 100 kHz, silently undoing the rate this project had set a moment earlier.

It hid almost perfectly. The library's own full-frame push raises the clock for the duration of its own transfer, so the old path really did run at 400 kHz and every number in the earlier documents is correct. Only the new partial flush, which does not go through the library's transfer, inherited the leftover 100 kHz:

Two nearly identical measurements from unrelated causes, which is why the arithmetic found this and the stopwatch did not. Passing the intended rate as both constructor arguments is what turned 23 ms into 1.8.

The rate is 800 kHz on the C6 and stays at 400 kHz on the classic board, which is proven at that rate and has no frame budget problem to solve. The panel controller's datasheet rates 400 kHz; the parts run well above it, but it is a bus, so whether 800 kHz holds depends on the wiring rather than on the part.

Where the CPU went: software floating point#

On this instruction set there is no float hardware. A multiply is a function call of a few tens of cycles; raising a number to a power is hundreds to thousands. The audio chain did exactly that once per sample.

Making the agent audible explains why the compressor exists: measured speech output peaks at 0 dBFS with its RMS 16 dB below, so it is faint on a small speaker even at full volume, and only compression closes that gap. Its gain curve raised the amount by which the signal exceeded the threshold to a fractional power — once for every sample.

At the voice's native 22,050 Hz that is 22,050 of the most expensive call on the part, every second of every reply, inside the same task doing the network reads for that same stream, and time-slicing against the renderer.

The exponent is constant for a whole reply, and the input is bounded. So the curve is now tabulated once when a reply starts and interpolated per sample: 512 evaluations a reply against 22,050 a second. The resampler's floating-point position became a fixed-point integer one on the same grounds.

What fixed point cost, and what it nearly cost#

The compressor and the resampler are deliberately free of the Arduino API so they can be tested on a development machine. The test keeps a copy of the floating-point implementation and holds the integer one to it, which is the only reason three separate errors did not ship:

The third is the instructive one. A right shift rounds downwards, so each step loses up to half a bit, and a one-pole filter with a 150 ms release has a long memory: the bias accumulates to roughly half a bit divided by one minus the smoothing coefficient. Rounding instead of truncating took it from 4.8 units to 0.11.

The final figures are a worst sample of 11 of 32,768 (−69 dBFS, one sample in 8,192), a mean error of about 1, and bit-exact output at unity settings — which matters because unity is the path a board with the compressor disabled takes, and a rounding error there would alter audio that was meant to pass through untouched.

Each of those three would have been called "close enough" by ear. None of them was found by listening.

The scanline fill#

With the bus no longer dominant, the renderer became the larger half of the frame. The 2.5D rig describes the even-odd scanline fill that draws every eye and mouth. It walked all 64 rows for every polygon, testing all 32 to 34 edges against each — roughly 6,300 pairs of floating-point comparisons a frame across two eyes and a mouth, on a part where a floating-point comparison is a library call.

Almost all of that work answered "no". The mouth is about ten rows tall and paid for sixty-four.

Each edge's row range follows from its two endpoints alone. The original test asks whether the two ends of an edge sit on opposite sides of the scanline, which is true exactly when the scanline falls between the edge's lowest and highest points — and that is a range of whole rows that can be computed once. Doing so bounds the row loop to the rows a shape actually covers, and reduces the per-row test to two integer comparisons. A horizontal edge yields an empty range, which is what the original did with one too, so the rows chosen and the intersection arithmetic are unchanged.

Four smaller items came out alongside it: two identical projection objects were built per frame from the same pose, the eye outline divided by the same constant on each of 32 points three times over, the sneeze effect reprojected a fixed pose every frame, and the C6 target moved from optimising for size to optimising for speed, which matters more than usual when every floating-point helper is out-of-line.

Together: 6.3 ms a frame down to 4.0 ms.

Why the face is provably unchanged#

The host test suite renders every 3D scene and compares pixels — 198,397 checks. Every change above passes it unaltered, so the face is bit-for-bit what it was.

That includes the one change not provably safe in advance: replacing a division with a multiplication by its reciprocal, which can differ in the last bit. The test settled it, not the argument for it.

Scheduling, once the costs were right#

The priority scheme is still necessary — it is simply not sufficient, which is what the first rounds confused. The underlying framework runs its own tasks far above the Arduino default, and on one core everything lands together:

Both audio tasks block waiting on the audio hardware rather than spinning, so a high priority costs the rest of the system almost nothing.

Two details matter more than the numbers. The Arduino core hard-codes its loop task at the lowest useful priority and offers no way to ask for anything else, so start-up raises it in place. And the loop returned immediately when no frame was due, spinning at the same priority as the task filling the audio buffer — two runnable tasks at equal priority share the processor, so the renderer was taking about half of it to re-read the clock. Sleeping for a single millisecond instead handed that half back.

Measuring any of this#

None of the above was measurable when it started, and the first attempts to measure it produced numbers that meant nothing. Two instruments were wrong in ways worth recording.

The frame-time counter was a since-boot maximum that never reset, so it permanently recorded the worst frame of the boot sequence — WiFi association, name resolution, the first clock sync. A reading taken afterwards said nothing about the steady state, and twice produced a confident report of a regression that had not happened.

The one probe that did separate render cost from transfer cost logged over USB serial. On this board a write blocks when the host is not draining it, which froze the device and was mistaken for a rendering fault — the probe causing the symptom it was added to find.

What replaced them, in the status endpoint:

All of them are a window since the last reset, not a total since boot. One request clears every window together, so a reading cannot straddle a change. The protocol is reset, wait, read — and only after the network has settled.

Frames over budget are counted against the budget plus one millisecond rather than the budget itself. The loop waits out the gap by sleeping, so it can only wake on a scheduler tick, and a perfectly on-time frame lands at 33 or 34 milliseconds. Counting 34 as a miss read 48% while the panel was holding 30 frames a second.

The reset reason exists because an unexplained reboot could previously only be guessed at, and the distinction that matters is not subtle: a crash or a watchdog is firmware, a brownout is the amplifier pulling the supply down and is wiring. They look identical from outside — the device comes back with a small uptime.

What it adds up to#

Measured on the board, 3D face, over reset windows:

The last row is the one that decides how it sounds: the producer never once outran the buffer, which is what the gaps in the voice had been.

Failure modes#

A buffer that competes for contiguous memory, failing two subsystems away. The wake listener keeps a little audio from before the trigger, so the first syllable is not lost. It was raised from 300 ms to 600 ms to recover the start of a follow-up begun while the microphone was still arming. The result was ten upload stalls in twelve voice turns, each answering "I couldn't hear you just now" because the speech-to-text upload ran dry while the turn was still open.

The extra 9,616 bytes took the largest contiguous free block from 167,924 down to 155,636. That is the same starvation audio diagnostics already records at 1,500 ms, where the board left the network entirely. The cost is not the 9 KB; it is that this buffer competes with the WiFi stack for contiguous memory, and the symptom surfaces as an audio upload stalling. Budget against the largest block, never against free memory, which still looks roomy once the largest block is gone.

A wake word the transcriber spells differently. The name is matched in the transcript, not in the audio, so a spelling the transcriber has not produced before is a rejection. One session rejected every utterance because the transcript read "Reboto" rather than "Boto" — the transcriber glues a fragment of the kept pre-trigger audio onto the front of the name. Nothing reached the model, the conversation history stayed empty, and the fault looked like the follow-up window rather than the wake gate. The voice endpoint reports what was heard, which answers this in one request; adding a spelling is a one-word edit.

An energy gate calibrated against the wrong floor. The wake threshold defaulted to about twenty times the measured quiet-room noise floor, so only speech directly at the microphone cleared it. It is a runtime setting, so it is tunable without a rebuild, but it has to be set against a measured floor rather than a remembered one.

A cooldown that is dead time. The microphone is muted while playback drains and for a cooldown afterwards, and that timer is pushed out again on every frame audio is still pending. At 900 ms the microphone armed roughly 820 ms after the sound stopped, so a follow-up asked immediately lost its opening words — one request for a joke reached the transcriber with its first word missing, and the model answered the fragment. Once the buffer is empty the only sound left is the hardware's own tail of about 80 milliseconds.

A model answering through a tool's text field. With the device's capability list declared, the cloud model answered "tell me a joke" by calling the animation tool and writing its words into that tool's spoken-phrase argument — whose description asked for one short sentence. It obeyed, so the reply was the set-up with no punchline. The same prompt with no tools declared told the whole joke every time. A description that constrains length constrains the answer, not a caption, whenever the model can route its reply through it.

What this leaves stale#

This document contradicts claims made earlier in the sequence, which are left in place rather than quietly corrected, because the reasoning behind them was sound for the board they were written for:

The frame-budget figure in runtime still shows the classic board's budget, and is correct for it.