Architecture: ESP32 I2S Audio Runtime#

Note: This document was generated by an AI agent.

Overview#

The robot-face firmware uses the ESP32's two I2S peripherals to capture microphone audio and play voice-agent replies without ever blocking the 30 fps face renderer.

• I2S0 is receive-only: an INMP441 MEMS microphone at 16 kHz.
• I2S1 is transmit-only: a MAX98357A class-D amplifier at 24 kHz.
• DMA moves samples between each peripheral and RAM; the CPU never touches a bit.
• Two FreeRTOS tasks do chunk-level work, both pinned to core 0, leaving core 1 to the Arduino loop that draws the face.
• The audio module is the only firmware boundary that exposes audio behaviour. Nothing else in the codebase includes an I2S header.

For the electrical rationale and wiring, see ESP32 Voice Hardware. For the render-loop constraint this architecture exists to protect, see Face rendering and effects.

---

What I2S Actually Is#

I2S — Inter-IC Sound — is a synchronous, fixed-rate, one-way PCM transport. It is not a bus in the sense that I2C or SPI are buses. There is no addressing, no command set, no acknowledgement, no flow control and no error detection. There is only a clock that never stops and a data line whose bits are audio by definition of when they arrive.

That single property explains almost every design decision below. If a sample is not on the wire at the instant the clock asks for it, silence is not what you get — you get whatever was in the buffer, at the wrong time, and it is audible. The bus cannot be asked to wait.

The three signals#

Data flows one way only, which is why capture and playback cannot share a peripheral here: the mic drives SD toward the ESP32, the ESP32 drives a different SD toward the amp. Each direction needs its own BCLK/WS pair too, because the two run at different rates.

One WS period is one stereo frame, always — left slot then right slot — even though both devices here are mono. Leaving a slot silent is a configuration choice, not the bus format changing.

Who generates the clocks#

Both channels — one for capture, one for playback — are configured with the ESP32 as the bus master.

The ESP32 drives BCLK and WS in both directions; the INMP441 and the MAX98357A are both slaves with no clock source of their own. This is what makes the sample rate a firmware constant rather than something the firmware has to discover — the configured input sample rate is not a request, it is what the ESP32 imposes on the microphone.

It also means the peripheral is free-running the moment the channel is enabled, whether or not any task is reading or writing. Both channels are enabled once during startup and never disabled.

The clock arithmetic, with this board's real numbers#

The bit clock is fully determined: BCLK = sample_rate × slot_bits × slots_per_frame. Standard (Philips) mode always clocks two slots per frame, even when the slot mode is set to mono.

Capture throws away three quarters of what it clocks: half is the unused right slot, and half of the populated 32-bit slot is precision beyond the 16-bit PCM format the speech pipeline wants. That waste is free — DMA discards it without the CPU seeing it.

Put a logic analyser on GPIO32 and you should measure 1.024 MHz on the nose. If you measure half or double that, the slot width or slot count in the I2S configuration disagrees with what you think it is, and every sample you read will be misframed.

Slots, mono, and why the left/right choice is wired and configured#

The INMP441 has an L/R strap pin. Tied to ground it drives the left slot and leaves the right slot high-impedance; tied to 3.3 V it does the opposite. The firmware must be told which slot to keep — the input channel setting selects the left slot when enabled, the right slot otherwise.

The input channel setting defaults to left, matching L/R tied to GND. These two facts — a solder decision and a firmware setting — must agree. If they do not, the firmware reads the slot the microphone never drives and the reported microphone peak level reports a flat, exact zero forever, with no error anywhere: the bus has no way to report "nothing was driven". A permanent zero peak is therefore the first thing to check against this pairing, before suspecting the microphone.

Playback is the mirror case. The MAX98357A is mono and mixes what it receives as (L + R) / 2. Setting the transmit side to mono mode makes the driver replicate each sample into both slots, so the mix returns the original value unchanged.

Bit alignment: where the 24-bit microphone sample actually sits#

The INMP441 produces a signed 24-bit sample, left-justified at the most-significant end of a 32-bit slot, with the low 8 bits undriven. So a 32-bit word read out of DMA looks like this:

bit 31                          bit 8         bit 0
|<------- 24-bit sample ------->|<- 8 unused ->|

Which is exactly what the capture task's conversion does, in three moves:

1. An arithmetic right shift by 8 bits on the signed 32-bit word, so the sign bit is replicated and negative samples stay negative. This recovers the full 24-bit value.
2. Gain is applied while all 24 bits are still present.
3. The result is narrowed from 24-bit range down to 16-bit range by dividing by 256.

A worked example. Suppose the microphone emits the 24-bit value 4660 (0x001234). On the wire, left-justified, DMA hands you 0x00123400:

raw                  = 0x00123400 = 1,193,984
raw >> 8             = 0x00001234 =     4,660    the true 24-bit sample
* MIC_GAIN (1)       =                  4,660
/ 256                =                     18    the 16-bit PCM sample

With the gain setting at 1 that is identical to a plain 16-bit right shift (0x0012 = 18), which is what this conversion originally did. That equivalence is the whole reason the divisor is 256 and not something rounder: a gain of 1 has to mean "unchanged behaviour".

Now the same sample with the shipped gain of 8:

(4,660 × 8) / 256    =                    145

Doing it the other way round — narrowing first, then multiplying — would give 18 × 8 = 144. The difference looks trivial on one sample and is not trivial on this part. The INMP441 is a low-sensitivity microphone: measured on real hardware, ordinary room ambience peaks near −50 dBFS and speech is only a little above it. At −50 dBFS the 16-bit sample is around 103 out of 32,767 — a signal living in the bottom seven bits, where every bit of resolution thrown away before gain is a bit you never get back. Gaining before narrowing recovers real resolution; gaining after only scales up what a 16-bit shift had already discarded. ESP32 Audio Diagnostics records how that was found.

The saturation is deliberate too: values above the 16-bit maximum are clamped down to it, and values below the minimum are clamped up to it.

Without it, a loud peak wraps: a value just past 32767 becomes a large negative number, and the waveform inverts mid-peak. That does not sound like distortion, it sounds like a crack or a click, and it is much harder to diagnose than honest clipping.

The gain setting buys headroom at the cost of a ceiling. With gain 8 the arithmetic saturates once the 24-bit sample exceeds 32767 × 256 / 8 = 1,048,544, which is one eighth of 24-bit full scale, or −18 dBFS at the microphone. Anything louder than that clips before it reaches the network. That is the trade the default makes: a microphone quiet enough that −18 dBFS is a shout is better served by resolution than by headroom.

I2S is not I2C#

The names differ by one letter and the buses have nothing to do with each other. This board runs both, simultaneously, on separate pins — which is exactly why the distinction is worth stating once, plainly.

The last two rows are the design constraint. Pushing a full 128×64 frame to the OLED takes about 25 ms of the 33 ms frame budget, and that is fine, because I2C is happy to be interrupted. I2S is not. If an audio task ever shared a core with that I2C transfer, playback would be starved for 25 ms at a time — two and a half chunks of silence — every single frame. Hence core 0 for audio, core 1 for the face.

---

The ESP-IDF Channel Model#

The firmware uses the newer ESP-IDF v5 standard I2S driver, not the legacy driver-install API. The newer model is built around channel handles, one per direction: a transmit handle for I2S1 toward the MAX98357A and a receive handle for I2S0 from the INMP441.

Setting one up is always the same three steps:

1. Allocate a channel on a numbered peripheral in a role. The two possible outputs are a transmit handle and a receive handle; leaving one unset means "this peripheral is one-way". Playback only asks for the transmit handle; capture only asks for the receive handle. That asymmetry is the whole reason the project can listen and speak at once without reconfiguring anything.
2. Apply the standard-mode configuration, which groups three concerns: the clock configuration (sample rate), the slot configuration (bit width, mono/stereo, slot mask) and the pin configuration. Every unused pin is explicitly marked unused — capture marks its output pin unused, playback marks its input pin unused, and both mark the master-clock pin unused, because neither part needs one.
3. Enable the channel. Clocks start. From here the peripheral is running whether or not anyone reads or writes.

Every one of those calls is checked, and a failure latches an error state with a human-readable stage name. The status report then surfaces a string like "configure capture I2S" rather than a number, and the web diagnostics show it directly.

Two peripherals, two directions, one header between them. The application controller reaches across the boundary to start and stop capture and to fire diagnostics; it never sees a channel handle.

DMA sizing, in milliseconds rather than bytes#

The driver allocates a ring of 8 descriptors, each holding 256 audio frames. The useful way to read those numbers is as time, because time is rate-independent and bytes are not:

That ring is the hard deadline. The capture task has 128 ms to come back for more data before DMA overwrites samples it never read; the playback task has 85 ms to refill before the hardware runs out of anything to transmit. Both tasks operate far inside that: 20 ms and 10 ms respectively. The margin is deliberately large because core 0 also runs the network stack, and a TLS handshake is not a polite neighbour.

In bytes, sizing from two slots per frame, the pool is roughly 16 KB for capture (8 bytes/frame) and 8 KB for playback (4 bytes/frame) — about 24 KB of DMA-capable RAM permanently committed. On a board where the recorder must budget against the largest contiguous free block, that is a real line item, not a rounding error.

One property worth knowing: the RX ring is always filling#

Because the receive channel is enabled for the life of the firmware, DMA keeps capturing and overwriting even while the capture task is idling with no consumer attached. The first read after capture starts therefore returns whatever DMA happens to be holding — audio from up to ~128 ms before the request. For a wake-word or push-to-talk turn that is harmless and arguably helpful; it is simply not a channel that begins at the moment you ask it to.

---

DMA Back-pressure Is the Task Clock#

Neither audio task has a timer, a delay(), or a rate calculation in it. Both are infinite loops that run as fast as the hardware allows — and the hardware allows exactly realtime. This is the most important runtime idea in the module, and it is entirely implicit in the code, so it is spelled out here.

Capture blocks until the data exists. Each read asks for 1,280 bytes — 320 32-bit frames — which DMA produces in exactly 320 / 16,000 = 20 ms. So the call returns every 20 ms, on the microphone's clock, with no scheduling logic anywhere. A 100 ms timeout is attached — five times the expected wait — so it fires only if the peripheral has genuinely stopped, in which case the error is logged and the loop retries.

Playback blocks until there is room. Each write hands over 480 bytes — 240 samples. The DMA ring drains at 24,000 samples/s, so it frees space for 240 samples every 10 ms, and the write returns at that cadence. The task's real pace is set by the amplifier, not by how fast replies arrive.

The consequence is a rule with no exceptions: the playback task always writes a full chunk. If the ring is short it pads with zeros and counts an underrun; during a diagnostic it fills the remainder with the 440 Hz tone instead. It never writes less, and it never waits for the network. Skipping a write would not "pause" playback — it would hand the amplifier stale DMA contents at full rate, which is a buzz.

---

Capture Path#

The capture task is the simplest of the two. It idles when no consumer is attached, reads a 20 ms chunk, converts it, measures it, and hands it on:

320 samples / 16,000 samples per second = 0.020 s = 20 ms
320 samples × 2 bytes                   = 640 bytes per chunk
                                        = 32 KB/s to the Agent

Per chunk it does three things beyond the conversion described above:

• Peak measurement. The tracked peak value holds the largest magnitude in the chunk, with the most negative 16-bit value special-cased to its positive magnitude, since negating it directly is undefined in two's complement. This is what the Voice screen's level meter and the device status endpoint report.
• Re-checking the flag. The capture-requested flag is tested again after the read, not only before it. A 20 ms read is 20 ms during which the consumer may have gone away; delivering into a stopped agent is the race this avoids.
• Handing off synchronously. Handing the samples to the agent runs on the capture task. Whatever the agent does with those samples — ring-buffer them for an HTTP uplink, run the VAD, resample them for the monitor — it does inside the 20 ms budget, on core 0. An agent that blocks here for longer than 128 ms loses microphone data.

There is no jitter buffer on this side and there does not need to be one: a local microphone on a local clock produces samples at a rate that never varies.

---

One Microphone, Several Consumers#

Starting capture enforces a single rule for everyone: one consumer at a time, refused at the boundary rather than negotiated by each caller.

A request from a different agent while one is already capturing is refused, unless the current holder is marked preemptable — in which case it is displaced silently and a message logs the preemption.

The one exception is the preemptable flag, which exists for exactly one caller. The wake-word listener holds capture for as long as the device is idle — which is nearly always — so without a yield rule every browser diagnostic would be refused for the life of the session. A preemptable holder is displaced silently; anything that must not be interrupted mid-sentence (a real voice turn) takes capture without the flag and cannot be displaced.

Three consumers live inside the audio module itself, all implemented as agent subclasses so they need no special path through the module:

The recorder agent carries the module's sharpest memory lesson. Its buffer — 1,000 ms × 16,000 samples/s = 16,000 samples = 32,000 bytes — is the largest single allocation the firmware makes, and a successful allocation is not sufficient evidence that taking it was safe: it checks the largest available contiguous heap block against a minimum headroom before committing, and refuses to record if there isn't enough left.

Starving the WiFi stack of contiguous memory does not fail at the allocation. It fails minutes later, as a device that has silently dropped off the network — taking with it the only channel you had for asking what went wrong. Budget against the largest contiguous free block, never against total free heap, which still looks roomy after the largest block is gone. ESP32 Audio Diagnostics has the full history.

---

Playback Path#

Replies arrive over TLS in bursts; I2S1 transmits at an unvarying 24 kHz. The ring buffer between them absorbs the difference.

4,000 samples / 24,000 samples per second ≈ 0.167 s = 167 ms
4,000 samples × 2 bytes                   = 8,000 bytes
240 samples per chunk / 24,000            = 10 ms per playbackTask iteration

The buffer is sized one slot larger than its usable capacity, because a ring needs one spare slot to tell full apart from empty; the usable capacity is the configured 4,000.

It is lock-free by construction rather than by luck: the producer only ever advances the write index, the consumer only ever advances the read index, each writes its own index after the samples it describes, and both indices are 32-bit aligned words, which are atomic on this chip. That is also why a full ring drops the newest samples rather than the oldest — evicting the oldest would mean the producer touching the read index, giving that index two writers and a real race. Dropped samples are counted, never dropped silently. In practice overflow means the model produced speech faster than realtime, so queueing a chunk returns a short count and the producer is expected to retry the remainder.

Each iteration of the playback task does four things in a fixed order, and the order matters:

1. Read up to 240 samples; pad with tone or zeros if short.
2. Measure the chunk peak — before the volume limit, because turning the amplifier down is the amplifier's business and should not close the character's mouth.
3. Resolve the lip-sync level (below).
4. Scale every sample by the playback volume setting and write.

Volume is a safety limit, not a feature#

Every outgoing sample is multiplied by the volume percentage and divided by 100 before it reaches I2S.

I2S has no volume concept — there is no gain field in a frame, because a frame is just a number. So attenuation has to happen in PCM, before transmission. The default of 70 was reached by ear (35 was inaudible across a room), and it is a ceiling protecting an 8-ohm speaker rated for 1 W from a MAX98357A that can exceed that at 5 V. It is deliberately a software limit and not the amplifier's GAIN strap: it can be changed without rewiring or a power cycle. Raising it means validating the power path and the driver's rating first.

Why the mouth is driven by marks, not by the outgoing samples#

The obvious way to lip-sync is to open the mouth by the loudness of what is being written to I2S. That was tried and it does not work, because the compressor deliberately flattens speech dynamics — that is how a small speaker got loud enough. A level taken after compression barely moves, and the mouth simply stays open for the whole reply.

So the producer declares the loudness before it compresses, through a small function that marks the level just before queueing each block.

Each mark is stamped with the running count of samples handed to the ring. The playback task tracks the running count read back out, and looks up the level recorded near that position. The mark therefore surfaces when the speaker actually reaches those samples — up to a ring's depth, about 167 ms, after it was recorded. A 64-entry level queue covers a full ring at the block sizes the speech client streams in.

Two fallbacks keep this honest. If no mark has arrived within 1,500 ms, the level falls back to the measured chunk peak — which is what keeps the diagnostic tone and the mic monitor visibly moving the mouth, since neither of those marks anything. And either way the result goes through an envelope filter (attack 0.25, release 0.82 per 10 ms chunk) so a syllable opens the mouth immediately but closes it gently, instead of flickering.

Clearing playback has to reconcile the two counters — resetting the queued count to match the played count — or every mark after a barge-in would be stamped ahead of where playback has reached. That reconciliation deliberately omits a lock: the playback task may advance its counter between the two updates, costing at most one chunk of mouth offset, which is not worth a mutex on the audio path.

---

The End-to-End Latency Ledger#

Every buffer between the microphone and the speaker, with what it costs:

The playback ring is the one deliberately tensioned number. It is sized from a jitter target, not from spare RAM — 167 ms is enough to ride out normal network variation, and 8 KB is what that costs. Growing it would smooth more jitter at the price of contiguous heap that the cloud TLS path needs, and of that much more delay between a reply being generated and being heard.

---

Feature Gates#

Two flags deliberately control different layers:

With the I2S flag off, the entire audio implementation is compiled out and every audio function becomes a no-op that returns false, zero, or a "disabled" string. Nothing calling into audio needs a guard of its own. Separating the gates is what let the UI, turn-taking and state transitions be built and tested before the microphone and amplifier existed — and it keeps host tests using the stub agent free of any I2S dependency.

---

Configuration Reference#

The values below are the tracked firmware defaults.

Peripheral and pins#

Rates, chunks and buffers#

Diagnostics and lip-sync#

---

Runtime Invariants and Failure Modes#

The renderer must not block#

Neither capture nor playback may run on the render loop. DMA transports samples; the two FreeRTOS tasks do chunk work at priority 2 on core 0, while the Arduino loop draws on core 1. Network operations stay behind the agent boundary. This is not a preference — a single OLED frame is ~25 ms of a 33 ms budget, and a blocking I2S write sharing that core would starve one side or the other on every frame.

Sample alignment is a wire-format contract#

Changing the slot mask, bit width, or alignment without updating the conversion produces capture that is quiet, clipped, or corrupted — and never an error. Treat the 8-bit right shift and the divide-by-256 narrowing as part of the INMP441 contract. The gain setting is the one term in that arithmetic safe to tune, and only inside the headroom the saturation leaves.

Playback must remain continuous#

Network jitter is normal; the ring absorbs it and underruns emit exactly one chunk of silence. Skipping a write, or blocking until data arrives, does not pause the amplifier — it hands it stale DMA at full rate.

Volume limiting is safety behaviour#

The playback volume setting governs the maximum power delivered to an 8-ohm, 1 W speaker. Validate the power path and the driver's rating before raising it.

The two gates stay independent#

Host tests using the stub agent must not require I2S initialisation, and enabling I2S must not imply that a voice-agent connection is configured.

Symptom table#

I2S reports nothing, so diagnosis is by ear and by counter. These are the ones actually encountered on this board:

Measuring it directly#

If it comes to a logic analyser, these are the values to expect on a healthy board:

• GPIO32 (MIC BCLK): 1.024 MHz continuous.
• GPIO33 (MIC LRC): 16 kHz square wave, 50 % duty.
• GPIO35 (MIC DOUT): data during the low (left) half only; the right half is undriven.
• GPIO26 (AMP BCLK): 768 kHz continuous.
• GPIO25 (AMP LRC): 24 kHz square wave.
• GPIO27 (AMP DIN): identical 16-bit patterns in both halves, because mono is replicated into both slots.

A clock at half or double the expected frequency means the slot configuration disagrees with the datasheet, which is also why every sample would be misframed.

---

Practical Extension Guide#

Change the microphone or its format#

1. Confirm slot, alignment, bit width and signedness from the datasheet — left-justified 24-in-32 is a convention, not a rule.
2. Update the slot configuration and the capture conversion together; they are one contract.
3. Recompute BCLK and confirm the part supports it.
4. Recompute chunk duration and the uplink byte rate.
5. Verify a representative positive, negative, near-silent, and near-clipping sample through the conversion.

Change the output rate or buffering#

1. Confirm the amplifier accepts the rate (the MAX98357A takes 8–96 kHz).
2. Recompute the playback chunk duration and the DMA ring's depth in milliseconds.
3. Size the ring buffer from an explicit jitter target, then check the resulting largest contiguous allocation on the device status endpoint.
4. Confirm an underrun still emits exactly one chunk of silence.
5. Resample at the producer, rather than reconfiguring I2S1 per reply — reconfiguring means disabling an enabled channel mid-conversation.

Add another audio backend#

Keep every I2S detail inside the audio module and preserve its public contract, so the application controller, the voice session and agent implementations stay transport-independent. A new consumer of the microphone should be an agent subclass, as the monitor and recorder agents are — that way it inherits the arbitration rule instead of needing its own.

---

Referenced Files and Interfaces#