How Do You Run a Fair A/B Test? Level Matching, Blind Trial Counts and a Check Order for DACs Like KSHF-02

A fair A/B test has one hard requirement: matched levels. When two devices differ by 1 dB, most people pick the louder one in quick switching, and that has nothing to do with sound quality. Match within 0.1-0.2 dB, then use blind trials at a sensible count, and only then can you separate a real difference from a preference.

01 Why does 1 dB decide the outcome?

Loudness discrimination around 1 kHz is roughly 1 dB, and quick switching makes it easier. 3 dB is obvious; 6 dB is close to doubling the perceived level.

Loudness bias also overwhelms tonal judgement. The same recording played slightly louder is often described as fuller and more detailed.

So the key move in an A/B test is not listening. It is matching levels.

02 How do I match to 0.1 dB?

Three options, ranked by precision: bench measurement, sound level meter, ears.

Bench: play a 1 kHz sine, measure AC voltage at the headphone output with a multimeter, then convert with 20log10(V1/V2). 0.50 V against 0.56 V is 0.98 dB; 0.50 V against 0.51 V is 0.17 dB.

Sound level meter: pink noise, microphone at ear height, one earbud against the mic at a time. Phone apps are accurate to about ±2-3 dB, which is coarse.

Ears: only when you have no tools. Adjust until you can just barely not tell which is louder.

Method	How tight	Limits	What it means
Multimeter on the output	Within 0.1 dB	Needs mV resolution, fixed load	Most reliable for headphone output A/B
Sound level meter at ear	±1 dB	Room noise and mic placement matter	Fine for speakers, marginal for headphones
Phone app	±2-3 dB	Mic and calibration vary	Only rules out differences above 3 dB
Ears only	±1-2 dB	Bias creeps in	Easiest way to mistake loudness for quality

Two voltage ratios cover most cases: 1.2 times is about 1.6 dB, 1.4 times about 2.9 dB.

03 How many blind trials do I need?

ABX gives you two options in random order, so guessing scores 50%. Too few trials and luck beats you.

Trials	Correct needed	p value	What it means
8	7	0.035	Bare minimum, weak conclusion
10	9	0.011	Common and manageable
12	10	0.019	Balanced
16	12	0.038	Steadier for near-threshold differences

Nine out of ten still leaves a 1% chance of pure guessing. Eight out of ten gives p around 0.055, which no longer excludes luck.

Trial count is not enough on its own. Same earbud, same insertion depth, same volume. Changing the tip can move bass by 1-2 dB, enough to swamp the difference you are hunting.

04 Fast switching or long listening?

Fast switching catches transient differences: tone, sharpness, bass impact. It also magnifies level differences.

Long listening catches accumulated impressions: fatigue, pressure, bass boom, comfort. It adapts quickly, and after 20 minutes everything sounds fine.

The two do not replace each other. Use fast switching to find a suspect, then long listening to see whether it persists.

05 Which variables ruin an A/B test?

In order of impact: level, fit, tips, temperature, earwax, expectation, order.

Order matters too. If A always comes first, B gets judged as smoother. Rotating ABBA and BABA removes part of that.

Expectation is the hardest to control. Knowing the price and brand pulls ratings toward the price. That is exactly why blind testing exists.

06 A worked example with KSHF-02 and KSHF-01

The two units are specified differently. KSHF-02 is 361 mW, PCM 32-bit/768 kHz, DSD256. KSHF-01 is PCM 32-bit/384 kHz, DSD256. Both are rated around 130 dB SNR with a noise floor near 1 µV.

Into the same load at the same output voltage, say 0.5 Vrms at 1 kHz, both sit far above the hearing threshold. What differs is noise floor and distortion, not a house sound.

Feed a high-impedance, low-sensitivity headphone and the power gap appears: at the same volume position KSHF-02 reaches a higher SPL while KSHF-01 may clip first. That difference is drive capability, not decoding character.

KSHF-02 also swings differently on its 4.4 mm balanced and 3.5 mm single-ended outputs, so fix one jack for the whole test. For how to read the numbers, see SNR and THD.

07 What do I do with the result?

Before buying, ask whether you can reliably pick the difference in a blind test. If not, choose on features, connectors and price. If yes, write down the trial count and conditions instead of trusting one impression.

With no instruments at home, use a phone app for a rough match and at least rule out differences above 3 dB. Anything smaller usually needs 16 blind trials to hold up.

For background, read headphone EQ, cables and sound and digital vs analog volume.

About the author

This article was written by the KEYSION audio content team. We set KSHF-02 and KSHF-01 to 0.5 Vrms at 1 kHz with a multimeter, ran 16 ABX trials on the same 32 ohm earbud, and repeated the run above 300 ohms. At matched level into 32 ohms most rounds ended in a tie. Into high impedance the power difference started to show. The p values here are one-sided binomial approximations; the sample is small and the result should not be generalised.