ESP32 · 02-face
Face rendering and effects
Note: This document was generated by an AI agent.
The character on the panel is not a picture of a face. It is 23 floating-point numbers, blended from one set of values to another, and redrawn from scratch 30 times a second. Nothing is stored; every frame is computed.
This page covers why it has to work that way, what the 23 numbers are, how the blend between them is shaped, and the order the drawing happens in — which turns out to be load-bearing.
Why numbers and not pictures
The panel is one bit deep: a pixel is on or off, with no grey and no alpha.
That rules out the obvious approach. Twenty stored bitmaps, one per expression, would give you twenty faces and no way to get between them — with no blending, the only transition available is a cut. A character that snaps between poses reads as a slideshow, not as something alive.
So the face is parametric instead. An expression is a set of target values, a transition is interpolation toward them, and the drawing code turns whatever numbers it is handed into pixels. The blend happens in the parameters, where floats are free, rather than in the pixels, where there is nothing to blend with.
The cost is that everything must be describable as a number. There is no way to hand-author a shape that the parameters cannot express. In practice that has been a feature: it forces each new idea to become a reusable dimension of the face rather than a special case.
Every face here is produced by the same drawing code from a different set of the same 23 numbers. These renders come from a port of the firmware's own drawing code, so they are what the hardware draws, not an impression of it.
The FaceState contract
Twenty-three fields in five groups:
| Group | Fields | What they control |
|---|---|---|
| Eye shape | eyeW eyeH eyeR eyePinch | Box size, corner radius, waist |
| Eye position | eyeX eyeY eyeSpread eyeTurn | Gaze, separation, depth |
| Lids | lidL lidR lidArc | Upper lids 0–1, and the lower carve |
| Brows and mouth | browY browTilt mouthCurve mouthHalfW mouthOpen mouthThick mouthY | |
| Whole face and effects | faceX faceY fxSweat fxCold fxHand | Shake, bob, and three set-piece intensities |
Why every field must be a float
The tween walks the struct as a flat array of floats. It does not know what any field means, and that is the point: adding a new dimension to the face costs one line in the struct and nothing else. No mixing code to update, nothing to forget.
The compiler enforces this contract, because breaking it would be silent. A
single int or bool member would be reinterpreted as a float, producing a
denormal or a NaN that propagates through every interpolation. A compile-time
size check catches the mismatch, and the host tests guard the behaviour.
This is the single most important rule in the face code: FaceState members are all floats, always.
Expression presets
A preset starts from the neutral defaults and overrides only what changes. Happy overrides six of the 23 fields; the other seventeen keep their neutral values.
| Field | Neutral | Happy | What you see |
|---|---|---|---|
eyeH | 36 | 18 | Eyes squash to half height |
lidArc | 0 | 11 | A lower carve turns them into an upward arch |
eyeY | 0 | −2 | The pair rises two pixels |
mouthCurve | 0 | 13 | Midline lifts above the corners |
mouthHalfW | 11 | 24 | Mouth more than doubles in width |
mouthOpen | 0 | 6 | Lips part |
All six fields move together across the transition. Nothing is sequenced; the apparent choreography is just simultaneous interpolation.
One number can carry the whole emotion
Angry and sad use nearly the same eyes and nearly the same frown:
| Field | Angry | Sad |
|---|---|---|
eyeH | 24 | 26 |
lidL / lidR | 0.30 | 0.22 |
mouthCurve | −9 | −10 |
browY | 8 | 6 |
browTilt | +7 | −6 |
The entire difference in feeling is the sign of browTilt — inner ends down reads as anger, inner ends up reads as sadness. Everything else is within a pixel or two.
Features are hidden without needing flags: neutral, happy and sleepy leave browY at −24, which is off the top of the screen. The renderer draws them anyway and they land outside the buffer. No special case, no branch.
Interpolation and easing
The engine keeps a source pose and a target pose, computes normalised progress k from 0 to 1, and blends between them. Linear k starts and stops at full speed, which looks mechanical, so k is passed through an easing curve first.
Four curves, all Arduino-free and host-tested:
| Curve | Formula | Character |
|---|---|---|
easeInCubic | t³ | Creeps, then accelerates |
easeOutCubic | 1 − (1 − t)³ | Fast away, settles gently |
easeInOutCubic | 4t³, then 1 − (−2t + 2)³ / 2 | The calm slow-fast-slow default |
easeOutBack | 1 + c₃(t − 1)³ + c₁(t − 1)² | Overshoots, then settles |
Cubing is what makes an ease-in feel slow to start:
t | t³ | |
|---|---|---|
| 0.1 | 0.001 | a tenth of the time, a thousandth of the distance |
| 0.5 | 0.125 | half the time, an eighth of the distance |
| 0.9 | 0.729 | now moving fast |
The overshoot constant is not arbitrary
easeOutBack uses c1 = 1.70158, with c3 = c1 + 1. That number looks like someone's magic constant. It is the solution to a specific requirement.
f(t) = 1 + c₃u³ + c₁u² with u = t − 1, c₃ = c₁ + 1
f'(t) = u(3c₃u + 2c₁)
= 0 at u = −2c₁ / 3c₃ = −0.41990, so t = 0.58010
f(0.58010) = 1.100004
Exactly 10% overshoot, at 58% of the way through the transition. Solve it in the other direction — what c1 gives a 10% overshoot? — and the answer is 1.70154. The constant is that requirement, rounded.
Each scripted expression chooses a curve through a snappy flag. Sharp reactions — happy, wink, surprise, anger, the glass tap, hiccups — take the overshoot. Sad, sleepy, suspicious and fogging take the smooth curve. That one boolean is most of the difference between a startle and a sigh.
Animation ownership and timing
The animation engine privately owns the current pose, tween start pose, target pose, current expression, script position, and temporary character cue. Its public surface remains intentionally small: starting it, advancing it each frame, moving to the next expression, picking a random one, and toggling the demo loop.
The engine first blends from the previous pose to the target. Normal changes use the easeInOutCubic curve; reaction-like script steps can use easeOutBack for a small overshoot before settling. The easing math is kept free of the Arduino API, so it can be compiled and tested independently.
After the tween, the engine applies procedural motion to the resulting state:
- Blinks temporarily close the lids.
- Breathing and talking vary the face or mouth without needing separate pose data for every frame.
- Expression-specific timelines add actions such as shivering, hiccups, sneezing, fog wiping, and tapping the glass.
The script supplies the expression order and its transition/hold durations when the demo loop is enabled. It is off by default: a manually selected face holds indefinitely. Timed sub-actions must likewise clamp to a resting result after their timeline ends, because a user may leave an expression on screen longer than its scripted hold duration.
Choosing a random expression selects an existing scripted expression without enabling the demo loop. Ambient mode uses this path, so it shares the established animation vocabulary rather than introducing a second animation set.
Temporary character-cue values take visual priority over the selected pose, but do not replace it. Clearing or expiring the cue returns the face to its previous expression.
Drawing, and why the order matters
A one-bit buffer has no alpha, so a lid cannot be blended over an eye. It is painted black on top of it. Two of the six steps are erasures, and that makes the order load-bearing: draw the mouth before the lid erase and the erase would take a bite out of the mouth.
Three shapes, one parabola
The same expression, 1 − t², does three different jobs. In each case t runs
from −1 at one edge to +1 at the other, so the term is 1 at the centre and 0 at
the extremes.
The mouth places each column on a parabola: the horizontal offset across each half of the mouth is squared and inverted, giving a shape value that is 1 at the centre and 0 at the corners.
Both the curve and the opening are scaled by that shape value. That second one is what makes the mouth look like a mouth: the lips meet at the corners and part most in the middle, rather than opening as a uniform slot.
The eye waist uses it upright, deepest at the centre.
The happy squint uses it inverted, deepest at the edges.
The off-by-one that is not a typo
Both the waist and the carve divide by span − 1 and W − 1 rather than by span and W. With N columns indexed 0 … N-1, dividing by N means the last column lands at t = (N-1)/N, slightly short of +1. The profile never quite reaches its endpoint, and against the right-hand end cap that leaves a one-pixel notch. Dividing by N - 1 puts the last column exactly at +1 so the shapes meet flush. On a 128-pixel-wide panel one pixel is visible, and the source comments say so in both places.
The eye is a rounded square, not a capsule
eyeR comes from the pose and is clamped to min(W, H) / 2 — the point where a rounded rectangle becomes a capsule. The resting character is a square with soft corners; a circle is eyeR = eyeH / 2, which is a per-expression choice, not the default. Surprise takes eyeR to 13 of a possible 21, so the eyes get rounder without becoming circles.
When an eye is pinched, the end caps deliberately use the full half-height rather than the pose's eyeR, because a waisted eye is a capsule by construction — otherwise pinching the middle would also shrink the ends.
Motion that never stops
Interpolation changes the face between expressions but would leave it frozen in between. Three procedural layers run underneath whatever pose is selected, so the character is never still.
Blinks drive lidL and lidR on a fast-down, slower-up curve. The blink lasts 160 ms, split at 0.35 — 56 ms closing, 104 ms opening — which is roughly how an eyelid behaves. Symmetric timing reads as a twitch; the asymmetry is what makes it read as a blink.
The scheduling matters as much as the curve. The next blink is normally 2.2–5.2 s away, but one time in four it is only 180–400 ms, so blinks arrive in bursts rather than on a metronome. The sleepy and wink expressions, and the sneeze, are excluded, since each scripts its own lids.
Talking sums two sine components at unrelated frequencies. One alone is visibly periodic — the mouth becomes a metronome. Two with an irrational ratio between them do not repeat on any timescale the eye catches.
Saccades move the gaze the way idle eyes actually move — in small involuntary jumps rather than a smooth drift. A new target is chosen every 0.7–2.2 s and reached over 220 ms on an easeInOutCubic curve, on both axes, so the gaze wanders instead of sweeping side to side. The range depends on the pose: neutral roves ±4 px horizontally and ±3 px vertically, while the look expression opens it to ±16 and ±7 — wider than it is tall, because real eyes are.
Breathing is a sub-pixel bob applied to the whole face:
faceY += sin(t × 1.8) × 0.9 period 2π/1.8 ≈ 3.5 s, amplitude 0.9 px
Under one pixel of amplitude is the point. Integer rounding makes the face hold still, shift by a single pixel near the peak, and return — read as life rather than as a visible bob.
Set pieces are timed impulses on top of everything else — the hiccup jolt on faceY, shivering on faceX, the sweat, cold, and hand effects gated by their fx* intensities. The sneeze is the fullest one, and its shape is typical:
| Elapsed | Behaviour |
|---|---|
| 0 – blast (1350 ms) | Wind-up: lids screw shut, brows drop, head tips back, tremble growing quadratically |
| Blast | The blast — spray, and cracks starting across the glass |
| → settle (2400 ms) | Recoil shake decaying toward zero |
| After settle | A sheepish resting face behind broken glass, which later repairs |
Each must clamp to a resting value when its timeline ends, because a user can leave an expression on screen far longer than its scripted hold. This is an invariant, not a nicety: a timeline function that keeps integrating past its story will drift a position until it overflows on a face nobody was watching.
What this costs per frame
The face gets whatever the panel transfer leaves — about 8.3 ms of the 33 ms frame, as the runtime accounts for.
Classic rendering is cheap: a handful of filled rectangles and circles, two Bresenham brow lines, and at most 2 × eyeW + 2 × mouthHalfW vertical line calls for the carves and the mouth. Every operation is integer, and none of it allocates.
A debug logging option reports peak draw and peak transfer separately over serial every two seconds, if you need to know the real number on your board. Measure before optimising; the answer is usually that the panel, not the drawing, is the cost.
Failure modes
| Symptom | Cause |
|---|---|
| Every interpolation produces NaN or garbage | A non-float member was added to FaceState |
| A new field never animates | It was added after the fixed field count was captured, or outside the struct |
| One-pixel notch beside a pinched eye | A divisor reverted from span - 1 to span |
| The mouth opens as a uniform slot | mouthOpen is no longer scaled by shape |
| Lid erase bites into the mouth | Draw order changed; the mouth must come after the erase |
| An expression stays mid-set-piece forever | A timed sub-action did not clamp after its timeline ended |
| Brows visible on neutral | browY no longer parked at −24 |
| The eyes look like capsules at rest | eyeR is being clamped to the maximum instead of taken from the pose |
Adding an expression or effect
For a pose-only expression:
- Add a new expression value to the pose model.
- Add its target values alongside the other presets.
- Add a step to the demo script if it belongs in the automatic sequence.
- Add or update host tests for the preset and interpolation behavior.
For an expression with a timeline or graphic effect, also add the timing and state adjustments to the animation engine, then implement the drawing primitive in the effects layer. Prefer representing the activation as a float field in FaceState when it must interpolate with the pose. Keep effects deterministic from the current time, so the frame loop can redraw them without stored rendering state.
Do not couple cloud-agent states, touch gestures, or screen navigation directly to the drawing or effects layers. Those callers select an expression or a character cue through the animation engine.
Test boundary
Host tests cover the pure parts of this design: easing curves, expression presets, FaceState interpolation, and script invariants. They do not execute the OLED drawing calls or ESP32 runtime because those depend on Arduino and the display driver.