Architecture: Making the Agent Audible#

Note: This document was generated by an AI agent.

Overview#

The device answers out loud through a speaker the size of a postage stamp: an 8 ohm driver rated at 1 watt, 16 × 9 mm. The synthesised reply arriving from Piper is, by the standards of that driver, both too quiet on average and already as loud as it can possibly be — a contradiction that has a specific cause and a specific fix.

This document is about the output side only. The input side — recovering the microphone's low bits with the gain setting — is a separate problem with a separate answer, covered in Audio diagnostics.

Every stage between the synthesiser and the air. Only one of them made a real difference, and it was not the one that looks like a volume control.

---

Decibels relative to full scale#

Everything below is stated in dBFS, so it is worth being exact about what the unit means, because two different quantities are measured in it and confusing them is the whole subject of this page.

A 16-bit PCM sample is an integer in [-32768, 32767]. Its level in decibels relative to full scale is

dBFS = 20 · log₁₀(|x| / 32768)

If that line needs unpacking — why 20, why the logarithm, why the values are negative — see Decibels, from scratch.

Three consequences follow from that definition and are used throughout:

• Zero is a ceiling, not a target. A sample cannot exceed full scale. There is no positive dBFS; arithmetic that would produce one produces a wrapped or clipped sample instead.
• A factor of two is 6.02 dB, and a factor of ten is 20 dB. So a microphone gain of 8 is +18.1 dB and a playback volume of 70% is −3.1 dB, without either needing to be measured.
• Power goes as the square of amplitude, so a change of x dB in level is a factor of 10^(x/10) in watts. The +8.9 dB this page is about is 7.8× the acoustic power, which is why it is audible across a room and the original was not.

Peak and RMS are not the same measurement#

The peak is the largest single sample in a passage. The RMS is the root of the mean square over it — the quantity that tracks power, and therefore roughly what loudness follows. Their difference is the crest factor:

crest factor = peak dBFS − RMS dBFS

A sine wave has a crest factor of 3.0 dB. Speech has a large one, because it is mostly quiet with occasional plosives and vowel onsets. A signal can therefore sit at the ceiling and still be faint, and that is exactly what arrives here.

Why a 16 mm driver is peak-limited before it is power-limited#

The 1 W on the speaker's label is a thermal rating: the average power its voice coil can dissipate without cooking. Excursion is a separate limit and a much nearer one — the cone has millimetres of travel, and an instantaneous peak that demands more than it has produces distortion immediately, at an average power nowhere near the thermal rating. The ledger below puts real numbers on that gap: the peaks reach about 490 mW while the average sits near 85 mW, less than a tenth of the rating.

So the loudness available from this driver is not bought by delivering more average power up to the peaks. It is bought by moving the average closer to a peak that is already fixed.

---

The Measurement That Framed Everything#

Before changing anything, the actual reply audio was measured:

Those numbers together describe the whole problem.

The peak is already at maximum. There is no headroom. Multiplying the samples cannot make them louder; it can only push the peaks past full scale, where they square off into distortion. Whatever a "volume up" control might feel like it should do, arithmetic will not do it here.

The average is 16.5 dB below that peak. A small driver can only express what is happening on average. Sixteen and a half decibels of range sat unused between the loudest instant and the typical one.

So the goal was never "make it louder". It was close the gap between the peak and the average, which is exactly what a compressor does.

Why not the amplifier's gain strap?#

The MAX98357A has a GAIN pin selecting between fixed gains, left floating for its default 9 dB. Raising it amplifies the same signal and the same silence, including the amplifier's own noise floor, and still cannot exceed the 1 W the driver will take. It moves the ceiling, not the gap. The gap is the problem.

---

The Compressor#

The compressor implements a conventional feed-forward, peak-sensing dynamic range compressor with a hard limiter after the makeup gain. It is free of the Arduino API so it can be exercised by a host-side test.

The gain law, derived#

Above the threshold T, the compressor allows only 1/r of the excess through. In the source that is a single power law: the gain is the ratio of the envelope to the threshold, raised to the power (1/ratio − 1).

Taking logs turns it into the textbook straight line. For a steady input at level in (so that the envelope has settled to it), with makeup M, all in dB:

in ≤ T:   out = in + M
in > T:   out = in + (1/r − 1)·(in − T) + M
              = in/r + T·(1 − 1/r) + M

Now substitute what this firmware actually sets. A compression threshold of 20% is 20% of full scale, and a makeup gain of 500% is a factor of five:

T = 20 log₁₀(0.20)          = −13.98 dBFS
M = 20 log₁₀(5.00)          = +13.98 dB
r = 4

out = in/4 + (−13.98)(0.75) + 13.98
    = 0.25·in + 3.49 dB

The static curve. The teal line is the gain reduction; the violet line is what actually reaches the speaker.

The threshold maps to exactly 0 dBFS: 0.25 × (−13.98) + 3.49 = 0.00. That is not a coincidence of the design so much as a coincidence of two independently chosen constants — 500 % makeup is precisely 20 % inverted — but it is the fact that makes this configuration behave the way it does. Every envelope at or above the threshold produces an output pinned at full scale, and the hard limiter, not the ratio, is what holds it there. At the source's 0 dBFS peaks the curve wants +3.49 dB and the limiter discards all of it.

What the ratio buys, given that#

If the region above the threshold is entirely the limiter's, the 4:1 ratio would appear to do nothing. It does, because the envelope is not the sample. The gain is computed from a peak follower and then applied to whatever instantaneous sample is passing, which is usually far smaller. For a sample at level s under an envelope e above the threshold:

out = s + M − (1 − 1/r)·(e − T)
    = s + 13.98 − 0.75·(e + 13.98)

So a whole syllable is scaled by its own peak: the ratio decides how much of its loudness a loud syllable gives back, and the makeup lifts everything underneath. That is the mechanism. Nothing about it is a volume control.

Checking the measured result against it#

The recorded outcome of these settings is +8.9 dB louder, with about 6 % of samples clipped. The makeup alone is +13.98 dB, so the compressor must have been applying, on average, 5.08 dB of gain reduction. Running the gain law backwards gives the envelope that corresponds to:

gain reduction 5.08 dB  ⇒  (e/T)^0.75 = 10^(5.08/20) = 1.799
                        ⇒   e/T        = 1.799^(4/3)  = 2.18
                        ⇒   e          = 2.18 × 6553.6 = 14,292 counts
                                       = −7.2 dBFS

An average envelope at −7.2 dBFS sits about 9 dB above the reply's RMS and about 7 dB below its peak, which is the crest structure of ordinary speech. The measured figure and the arithmetic agree, which is the point of doing both.

The clipping follows from the same numbers. With the envelope at −7.2 dBFS the net gain applied is +8.90 dB, so any sample above

32767 / 10^(8.90/20) = 11,761 counts = −8.9 dBFS

is folded onto the ceiling — that is 2.40× the RMS. A Laplacian amplitude distribution, the usual model for speech, puts about 3.4 % of samples beyond 2.40 σ. The measured 6 % is higher because clipping is not spread evenly: it concentrates in the loud passages, which are exactly the ones whose envelope is highest. Six percent is the price of the loudness, and on speech through a 1 W driver it is not perceptible. Refusing to clip at all would have cost most of the 8.9 dB.

The peak barely moves. The average rises to meet it.

The envelope follower#

The gain above is a function of an envelope, and the envelope is a one-pole smoother over the rectified signal that moves toward the incoming level each sample — fast (using the attack coefficient) when the level is rising, slow (using the release coefficient) when it is falling.

The coefficients come from the standard exponential form, which the compressor's coefficient calculation states directly:

c = exp(−1 / (τ·fs))      with τ in seconds
  = exp(−1000 / (ms·fs))  as the source writes it

Piper's faber voice returns a 22,050 Hz WAV, and the compressor is started on the WAV's own rate rather than the output rate, so the time constants hold whatever voice is selected — edresson at 16,000 Hz gets different coefficients and the same 5 ms and 150 ms:

Those two numbers are chosen against two different features of speech:

The attack is faster than a syllable and slower than a waveform. 11.5 ms to settle means the first ~11 ms of a plosive passes at more gain than the steady state, and lands in the limiter. That is intended: catching it fully would require an attack short enough to distort the waveform itself.

The release is longer than a pitch period, which is what makes this a syllable follower rather than a waveform follower. A male voice at 125 Hz presents a glottal pulse every 8 ms; across that gap the envelope falls by only

20 log₁₀(exp(−8/150)) = −0.46 dB

so the gain barely moves within a voiced sound, and the compressor rides syllables rather than modulating the vowel it is inside. A release of, say, 5 ms would instead track individual pitch cycles and audibly distort.

The envelope state persists across blocks. Audio arrives in chunks from the network, and a gain envelope that restarted each chunk would pump audibly at the chunk rate. The compressor instance is therefore long-lived and reset once per utterance, for the same reason the ring modulator keeps its carrier phase.

---

The decibel ledger#

Where the 16.5 dB goes, from the synthesiser to the voice coil. Full scale at the amplifier is 2.83 V RMS into 8 ohms — that is the 1 W derived in voice hardware — so a dBFS figure converts to volts and to watts directly.

The last row is the argument of this page in two numbers. Compression raised the average power the driver receives from 11 mW to 85 mW — 7.8× — while the peak power did not move at all, because the peak had nowhere to go. The thermal rating is never approached from either side; the excursion the peaks demand is what sets the ceiling, which is why closing the gap was the only lever available.

---

The Stages That Did Not Help#

Playback volume scaling is a plain multiplier applied last, per sample, saturating rather than wrapping. It is a limit, not a boost — useful for turning the device down, useless for making it louder, because the signal reaching it already peaks at full scale. It is deliberately a software limit rather than a hardware strap, so it can be changed without rewiring and cannot be forgotten in the hardware.

The ring modulator ships disabled (depth at 0%). Ring modulation was built as a character effect and measured before being switched off: multiplying by a sine carrier costs 2–3 dB of average level, because the carrier's zero crossings silence the output periodically. That is the exact opposite of what this speaker needs. The gain follows (1 − depth) + depth × carrier, so the knob is continuous — depth 0 is a constant gain of 1 rather than a second code path — and the code stays, documented, for anyone who wants the effect more than the volume.

---

The Cost Nobody Predicts#

Compression works by removing dynamic range. That is not a side effect — it is the mechanism. And the lip-sync animation was reading exactly that range.

The same speech before and after. A mouth driven from the right-hand panel has almost nowhere to move.

The Speak panel and the idle face open the character's mouth from the amplitude of what is playing, computed as the playback level against a reference magnitude for a fully open mouth. Measuring that amplitude in the playback task — where the samples go to I2S — seemed obviously correct: it is the audio that is being heard right now, needing no timing compensation.

It was correct about timing and wrong about content. Post-compression speech is nearly constant, so the mouth pinned wide open for the entire reply and shut when it ended. Correct timing on the wrong signal.

Measuring early, and carrying the measurement forward#

The only place the speech still has its own dynamics is before the compressor — which is also up to about 167 ms before anyone hears it:

VOICE_PLAYBACK_BUFFER_SAMPLES 4000 ÷ VOICE_OUTPUT_SAMPLE_RATE 24000 = 166.7 ms

The speech client therefore takes each block's peak before compression runs and marks the level. The level queue stamps that measurement with the sample position it describes and holds it until the playback task reaches that position.

The alignment is exact rather than tuned, and the reason is worth keeping: the ring's occupancy is the offset. A running sample counter on each side is all it takes — no guessed delay, no drift. The block's peak is marked before it is queued, so the stamp lands on the position those samples will occupy in the ring, and only then does the block pass through the compressor.

The queue is sized from the same arithmetic. The speech client reads into a 256-entry buffer, so one mark covers at most 256 source samples, which resample to 256 × 24000/22050 ≈ 279 output samples. A full ring therefore holds 4000 ÷ 279 ≈ 14.4 marked blocks, and the 64-entry level queue is over four times that. Marking drops the oldest entry rather than refusing the write, because a consumer that far behind needs the marks nearest the present, not a frozen mouth.

Two smaller decisions round it out. Audio that nothing marks — the diagnostic tone, the microphone monitor — falls back to measuring the outgoing samples, which is all the shape it has; a 1,500 ms validity window is how long a mark stream stays authoritative before that fallback resumes. And speaking a reply marks a zero at the end of each utterance so the mouth closes on the last sample rather than holding the final syllable.

Smoothing the mouth#

A raw block peak jitters far too much to drive a face, so an envelope filter smooths it. That filter deliberately knows nothing about sample rates: its coefficients are per-push retention fractions, and one push is one playback chunk — 240 samples at 24 kHz, exactly 10 ms. Converting the configured fractions to time constants:

τ = −Δt / ln(c)

attack  0.25  →  τ = −10 / ln(0.25) =  7.2 ms   (to 90 %:  17 ms)
release 0.82  →  τ = −10 / ln(0.82) = 50.4 ms   (to 10 %: 116 ms)

A syllable opens the mouth inside two frames of the 33 ms render loop, and it closes over about four — visibly a mouth rather than a flicker. The envelope filter also snaps anything below 0.5 to zero, because an exponential decay otherwise leaves a permanent sliver of a gap; from a fully open 20,000 that floor is reached after 53 chunks, so a mouth left with no further marks takes about 534 ms to shut on its own. In practice the explicit zero mark closes it far sooner.

Verifying it#

Because a mouth is hard to judge by eye and impossible to judge remotely, the playback level is exposed on the device's audio status endpoint. Sampled 120 times through a four-sentence reply:

Sentence gaps appear as clean decays to zero (502 → 235 → 142 → 0) followed by a jump back on the next word. Before the fix, the same trace would have been pinned near maximum throughout.

---

Failure modes#

---

Configuration reference#

To disable compression entirely, set the makeup to 100 with the threshold at 100; both are needed, since either alone still changes the signal.

---

Files Referenced#

See Also#

• The conversational agent — how a reply gets produced in the first place
• Voice hardware — why the speaker and amplifier are what they are
• I2S audio runtime — the playback task and the ring buffer
• Audio diagnostics — capture gain and the browser test surface