Architecture: ESP32 Audio Diagnostics and Test Surface#

Note: This document was generated by an AI agent.

Overview#

The two documents beside this one describe what the audio hardware is (Voice hardware) and how the firmware keeps it fed without stalling the renderer (I2S audio runtime). This document covers the third problem, which only became visible once real hardware arrived: proving that any of it actually works.

That turned out to be the hard part. A microphone can be wired correctly, clocked correctly, and decoded correctly, and still produce a recording that sounds like silence — because the part is quiet, because the firmware discarded the low bits, or because the diaphragm was destroyed by a cleaning solvent hours earlier. Each of those failures presents identically to the ear. The diagnostics described here exist to tell them apart.

Three checks, in the order they should be used: play a tone to prove the output path, measure the microphone to prove the input path, then loop one into the other only once both are individually trusted. The rest of this page is why that order, and not another.

---

What a measurement proves, and what it does not#

One failure during bring-up cost more time than every other combined, and it is worth stating as a general rule before the specifics, because the rule is what transfers.

A microphone stopped working. Every wire still passed a continuity test, so the investigation went looking at wiring — for some time. The multimeter was not merely unhelpful; it was actively misleading, because it answered a question nobody had asked:

Continuity proves that an electrical path exists. It says nothing whatsoever about whether the transducer at the end of that path still responds to sound.

Generalised: every instrument answers exactly one question, and a reading is only as useful as the number of distinct failures it can separate. A test that passes under both the healthy case and the failure you are hunting has told you nothing, however confidently it reads. So the useful property of a diagnostic is its specificity — how many causes survive it.

Three working rules fall out of that table, and every design decision on this page follows one of them.

Test the thing with the fewest dependencies first. A tone involves the amplifier, the speaker and one I2S peripheral. The loopback involves all of that, plus the microphone, plus a resampler. A failure in the first localises immediately; a failure in the last localises nothing. This is why the loopback is last despite being the most satisfying test to run.

Measure numbers, never impressions. The ear cannot separate "no signal" from "signal 50 dB down", and those two have completely different fixes. This is the entire reason the browser meter reads in dBFS rather than percent, and why bring-up inspects a recording's sample values rather than playing it back.

Prefer a reading that a healthy system cannot also produce. The underrun counter fails this test and is described below as the trap it is.

---

Capture Gain: Why the Microphone Sounded Broken#

The INMP441 is a low-sensitivity part, and nothing in its datasheet prepares you for how little of the available range ordinary sound occupies. Measured on real hardware at unity gain, a normal room's ambience peaked near −50 dBFS and speech sat only slightly above it. Played back through a computer's speakers, that is indistinguishable from silence with a noise floor — and it reads as a broken microphone when the microphone is working perfectly. The level ladder derives that figure from the datasheet, so it was predictable before it was measured.

The fix is not to amplify the result. It is to stop throwing away the bits before amplifying.

The microphone delivers 24 bits left-justified inside a 32-bit slot. The original conversion took a 16-bit right shift, keeping the top 16 bits of the sample — which is correct arithmetic and quietly discards eight bits of genuine signal. Applying gain afterwards can only scale up what survived that truncation, including its quantisation error.

The current conversion keeps the whole sample first: it shifts right by 8 to preserve all 24 bits, multiplies by the gain setting, divides by 256 to narrow to 16 bits, then clamps the result to the 16-bit range before storing it.

Two properties are load-bearing. The divide-by-256 is chosen so that a gain of 1 reproduces the old 16-bit-shift result exactly, which makes the change provably non-destructive at its default. And the result saturates rather than wraps, so an over-loud peak clips the way audio is expected to instead of inverting into a crackle that sounds like a hardware fault.

Both rows are the same quiet room, minutes apart, and the 18.9 dB between them is the 18.1 dB that 20·log₁₀(8) predicts, within the measurement's own noise. The gain is a plain multiplier rather than a shift so intermediate values are available; it should stay at or below 64, since the intermediate is a signed 32-bit value and a louder part could overflow it.

---

One Capture Stream, Multiple Consumers#

The agent interface is the firmware's provider-neutral audio sink: starting capture registers one, and the capture task delivers every chunk to it. The diagnostics are implemented as agent implementations rather than as special cases inside the capture path, which is why none of them required touching the capture task.

Only one may run at a time, and that rule lives in exactly one place: a request is refused outright if audio isn't ready or no agent was given, and refused again if a different agent is already capturing — unless the current holder was marked preemptable.

The capture-requested flag is the whole guard. Without it, starting a recording while the loopback was running would silently repoint the active agent mid-stream and corrupt both. With it, the second caller is refused and the first is untouched — and every future consumer inherits that protection without re-implementing a check.

The preemptable flag is its single exception, added for the wake-word listener. That listener holds the microphone for as long as the device is idle, so under the original rule every browser test would be refused for as long as the feature was switched on. It takes capture marked preemptable; the monitor and recorder displace it rather than failing, and the application controller's per-frame reconcile re-arms it once they are finished. Nothing else sets the flag. A forwarding wake turn still uses this registration, however, so avoid starting diagnostics during a turn — this is preemption, not a complete active-turn exclusion policy.

Two details of the capture task make the diagnostics cheaper than they look. The tracked peak value is updated before the agent call, so any consumer — including one that discards everything — is enough to drive a live level meter; a meter does not need the loopback, and therefore does not risk acoustic feedback. And handing samples to the agent is always done from the core-0 task, never from an HTTP handler, which is what keeps the web surface non-blocking.

---

The Recording Buffer#

The recorder holds one clip at a time. The buffer is allocated on demand when a recording starts, freed either when the browser fetches it or when a newer recording replaces it. Starting a second recording discards an unfetched first one, which avoids both a queue and a "you must fetch before recording again" state machine for what is a bench tool.

Its size is the most-revised number in the firmware, and the revisions are the evidence for a rule that is easy to state and easy to get wrong.

Why 1000 ms, arithmetically#

AUDIO_TEST_RECORD_MS 1000 × VOICE_INPUT_SAMPLE_RATE 16000 / 1000
                                        = 16,000 samples
                                        = 32,000 bytes at int16

16,000 samples ÷ VOICE_CAPTURE_CHUNK_SAMPLES 320 = 50 chunks exactly

A whole number of 20 ms capture chunks means the recorder agent's chunk handler never has to reason about a partial chunk at the end of the buffer — a safety clamp on how much it copies exists for safety, not because the arithmetic ever needs it.

Free heap is the wrong number#

A 3000 ms clip is 96,000 bytes, and the board reported roughly 119 KB free — apparently comfortable. The allocation failed anyway, because free heap is the sum of all free blocks while a single allocation needs one contiguous one. The device status endpoint reports both, as free heap and the largest contiguous block, and the largest contiguous block is the one to budget against.

A successful malloc is not the same as being safe to allocate#

This is the more expensive lesson. The recorder's buffer is the firmware's largest single allocation, so holding it starves whatever needs contiguous memory next — which on this board is the WiFi stack. That does not fail at the call site. It fails minutes later as a device that silently stopped answering, having dropped the only channel you had for asking it what went wrong.

The third row is the one that changed the design. Nothing about adding a wake word touched the recorder, and the recorder broke anyway — which is the argument against trusting a constant to stay correct as the firmware around it grows. So the check moved into code: starting a recording reads the largest available contiguous heap block after allocating, and hands the buffer straight back when less than the minimum headroom (16,000 bytes) remains. If that check fails, the buffer is freed immediately and the recording is refused.

The reset routine frees the buffer itself, so the refusal calls it and nothing else. That is worth stating because it briefly did both: an explicit free ahead of the reset left the same non-null pointer freed twice, which made the one path built to survive memory pressure the one that crashed.

The fetch handler was hardened alongside it. It abandons the stream when the client disconnects and frees the buffer unconditionally, because the download used to be the only path that released it — and the download is exactly what memory pressure breaks. A discard endpoint exists for the same reason: a way out that moves no data.

Every duration on this surface is a compile-time constant rather than a request parameter. A client can start a test but cannot choose how long the device spends on it, so no request can ask the board to record arbitrarily long or hold a test open indefinitely.

---

The contiguous-heap ledger#

The scarce resource on this page is not RAM but contiguous RAM, and the distinction has a consequence worth stating: static memory and the heap fail differently. A static buffer is carved out of DRAM before the heap exists, so it lowers the ceiling once, at link time, and can never fragment what is left. A heap allocation of the same size takes its block from the middle of the pool, and when it is freed at a different time from its neighbours the ceiling does not necessarily come back. That is why the wake pre-roll is deliberately static and the recorder's buffer deliberately is not: the pre-roll is held forever and the clip is held for a second.

One claimant left the ledger entirely, and it is the largest movement in it: cloud replies used to be fetched over HTTPS directly from the board, costing roughly 43 KB of heap per request. Routing them through the LAN proxy removed the TLS arena altogether, and free heap during a request rose from about 44 KB to about 93 KB — see the local pipeline. Two comments in the tree still describe the pre-proxy world and the pre-1000 ms buffer; the numbers in this table come from the code as it stands.

---

The HTTP Surface#

Every route sits behind HTTP Basic authentication, and the mutating ones additionally check Origin against http:// plus Host when Origin is present, rejecting a cross-site replay of credentials the browser has already cached.

The shape of this surface is dictated by one constraint: an HTTP handler may not block for more than about 50 ms, because handlers run inline on the loop task between rendered frames. A one-second recording cannot happen inside a request.

So recording is split: a POST that returns immediately, capture continuing on the core-0 task that was already running, a client poll against the status route the page was polling anyway, and a second request for the bytes. The same reasoning produces the monitor's safety stop — a loopback abandoned by a closed browser tab is stopped by the device after a fixed timeout, because the client that should have stopped it is gone.

The WAV response streams a 44-byte header followed by PCM in 2,048-sample chunks — 4,096 bytes, eight of them for a full clip — with a brief yield between each. No encoder is involved: the ESP32 has neither the cycles nor a library for MP3 or AAC, while a WAV header is 44 bytes of arithmetic and plays natively in every browser. Because the duration is fixed, the header's length fields are known before the first byte is sent, so the response can declare a Content-Length and needs no chunked-transfer machinery.

---

The Browser Page#

The Audio Test tab is the operator console for all of the above. It lives inline in the device's single-page management UI, which may not reference any external script, font or stylesheet — the page has to work when a board in access-point mode is the only thing reachable.

• The tab reveals itself by probing the status endpoint and hiding on failure, rather than inferring from a firmware version string what a given build contains.
• The meter reads in dBFS, not percent. Twenty segments span a floor of −60 dB up to 0 dBFS, so one segment is 3 dB — a factor of two in power, per block. The top two segments (0 to −6 dBFS) render red and the four below them (−6 to −18) yellow, so clipping is visible rather than merely full. A percentage meter was the original implementation and it was actively unhelpful: it made a −50 dBFS signal and true silence look nearly identical, which is precisely the distinction bring-up needs.
• Polling is a self-rescheduling chain, not an interval. Each response schedules the next poll 100 ms later, so a slow device cannot accumulate overlapping requests. The chain starts when the tab is shown and is cleared when it is left.
• The recorded clip is fetched with fetch() into a Blob, then attached to the <audio> element as an object URL. Assigning the route directly to <audio src> does not reliably reuse the Basic-Auth credentials the browser holds for the rest of the page — a media element's own network request is a different credential context — and it fails silently when it doesn't, which presents as a player that loads nothing and reports no error.

One readout on that page fails the specificity rule above and is worth knowing about: the underrun counter is not an error count. The playback task free-runs at roughly 100 chunks per second and records an underrun whenever the ring buffer had nothing queued, which at idle is always. The number therefore climbs continuously on a perfectly healthy device — it is a reading a healthy system also produces, and so it separates nothing. The drop counter — samples the ring buffer refused for lack of room — is the half that indicates a real problem.

The playback level is the other readout worth understanding. It is the number the on-device Speak tab opens the character's mouth with, and it is exposed here because a mouth is hard to judge by eye and impossible to judge remotely; polling it through a spoken reply is what distinguishes a mouth that articulates from one that simply sits open. It is not the loudness of the samples going to I2S — those have been through the compressor, which flattens speech by design. See making the agent audible for the measurement that is carried across the playback ring instead.

---

Always Listening#

With wake word listening on, the microphone is open whenever the device is idle. The full design is in the conversational agent; two of its interactions belong here because they change what the diagnostics do.

The listener yields. It takes capture marked preemptable, so the monitor and the recorder displace it rather than being refused, and the application controller's per-frame reconcile re-arms it when they finish. Without that flag, every browser test would be refused for as long as the feature was enabled.

The listener is deaf while the speaker is live. The microphone hears the amplifier, so Boto saying its own name would wake Boto; the cooldown is pushed forward for as long as anything is playing. A loopback test started while a reply is in flight is therefore testing something other than what it appears to.

---

Equivalent on-device panels for the mic and speaker checks, so those could be run from the OLED and the touch pad with no browser at all, were specified but superseded: both chips were proven by other means, and the Speak tab is the one on-device panel that came out of that spec.

---

Bringing Up New Audio Hardware#

The order below is the specificity rule applied — fewest dependencies first — and it reflects where time was actually lost, not an idealised procedure.

1. Wire the parts, then enable the audio build flag in the local configuration header and flash over USB.
2. Read the reported audio status. ready means both I2S peripherals initialised. It says nothing about whether either transducer works.
3. Test the speaker first, with the tone. It has no dependency on the microphone, so a failure localises immediately.
4. Test the microphone by measuring, not by listening. Record a clip and inspect its actual sample values — peak and RMS. Judging a quiet microphone by ear conflates "no signal" with "signal too quiet to hear", and those need completely different fixes. Raise the microphone gain until speech sits comfortably above the noise floor without clipping.
5. Use the loopback last. It exercises both paths simultaneously and therefore tells you least about which one is at fault.

Never clean a MEMS microphone with isopropyl alcohol, and keep flux away from its acoustic port. Unlike every other component on the board, the microphone has an open port leading directly onto a micron-thin diaphragm. Solvent wicks in by capillary action and can silence it permanently, and liquefied flux does the same. One unit was destroyed exactly this way during bring-up: a sudden ~33 dB drop to a flat noise floor, while every wire still passed continuity. Mask the port with Kapton tape during any rework, and use no-clean flux.

---

Failure modes#

---

Configuration Reference#

These are the settings this document adds to the ones already listed in the I2S runtime.

---

Referenced Files and Interfaces#

The components referenced below belong to this firmware. The full API catalogue is in web management and API.