How many bits are really meaningful?

[Mike Lavigne]

The Problem with A-B'ing and Why Neil Young is Right about Sound Quality.
by Allen Farmelo on April 4, 2012 - Comments (26)



With news of Neil Young pushing to get studio quality audio out to the consumer, it occurred to me that it would be good to get some perspective on how we go about comparing audio quality. One of the most problematic issues surrounding audio quality and file resolution these days has been that, more and more, we hear about people not being able to tell the difference between compressed audio and full-resolution audio, and by extension some people are claiming that efforts like Neil Young's are out of touch and irrelevant. I couldn't disagree more.

In my opinion, one of the biggest pitfalls in this arena is the dreaded A-B test. In our seemingly infinite desire to manipulate the response of our fellow humans, we tend to stage A-B tests in two ways:

1) Almost-Barely: A-B because you can barely tell, and thus proving that the two items are close enough to be interchangable.

2) Absolutely-Boldy: A-B because you want to show how different two things are (and typically the superiority of one of them, which the test-designer is likely selling).

In my opinion, both types of A-B tests won't tells us what we want to know, but it's the first type, the Almost-Barely tests, that I take issue with here. (And I'm also inadvertently and respectfully drawing into question what, Ethan Winer, (who I adore) has said about "scientific" A-B testing in Tape-Op #88, p.66.)

We've all done it at some point. A. B. A. B. Flip. Flip. Flip, flip, flip. This is B? Ok, A. B. Ok, is that B? Ok, do A? Gosh, they're close!

The Almost-Barely tests seem so objective. Here's A, and here's B. See, soooo close! The conclusion: because most people can't tell the difference, there's not really a difference. And then you stack up the stats on these results, and suddenly we're doing science.

The problem is that these tests assume that because two things are close enough in a quick test that the difference will also be indistinguishable over long stretches of time. However, this assumption totally misses how it is that we tend to actually experience things in our very real lives.

For example, we hear people talk about how one can't make out the difference between a hi-res MP3 and a 24bit WAV file (assumedly a difference similar to the one Neil Young feels is worth fighting for). Admittedly a hi-res MP3 and a 24bit WAV are relatively close enough in resolution that many people will not be able to pick them out in an A-B test.

But, we don't live with music like that. If your'e anything like me, you listen to a lot of music in a lot of styles and - over the course of, say, a month - perhaps you've absorbed well over a hundred listening hours across many different albums on a few different playback systems.

How can flip flip flip replicate what it is to live with that much music for that long? How can a drop of water emulate what it is to swim for hours?

If you want to do a real test of the differences, give people a music collection that's all MP3s for a month, then give them that same collection as 24bit WAVs for a month, and then ask which one's which, and I bet you will start to get some correct answers.

Why? Part of the answer is that, if given enough time, subtle differences will reveal themselves to us. Subconsciously at first, and eventually consciously, we become aware of new details, subtleties, nuances. We humans need time to truly come to perceive things in full detail. But details, once revealed, become important features in the big picture.

For example, when I started living with my partner I introduced her to what I call "good coffee." At first she kind of shrugged it off as my snobbery at work, and she couldn't really taste the difference. But then, after months of drinking the good stuff, she found herself to be a bit of a coffee snob, too. She could taste the difference because she had, simply, spent time with the good stuff. The coffee revealed itself to her, slowly and subtly. Her palate developed. And the thing about good coffee is that it holds more detail, nuance and, therefore, interest.

But it takes a while to become aware of that depth and complexity. Had she done a flip-flip-flip A-B and made her choice to only drink the cheaper stuff because, "you know, they're basically the same," she'd have missed an opportunity to develop her palate.

I think the same thing can be said for the resolution of music, and it breaks my sonic heart to think of the A-B tests out there designed to convince someone that because they can't tell the difference today they won't tell the difference in a month or a year. A-B tests may be designed to show that subtle differences don't matter, but what they really do is shut down the possibility that those subtle differences could be the key to someone's aesthetic awakening.

If you can't tell the difference when you flip flip flip between two subtly different sounds, please know that I'm here cheerleading for you to slow down and be a real human being. Live with the better quality for a while, and see if over time you too start to hear things you never heard before. There is nothing like real life to truly test the quality of something.

So, for this reason, I herald Neil Young (as I always have, come to think of it) for sticking to his beliefs and making efforts to bring affordable and easily accessible hi-resolution audio to the market. It's time we left the MP3 behind and opened back up the possibility of people developing their ears and loving great sound once more. Who knows where such high standards could lead us.

Thank you, Neil.

The aesthetic revolution will be beautiful!

Allen Farmelo
www.farmelorecording.com

amen.

 

[fas42]

Excellent train of thought by Allen ... thanks for linking, and posting it, Bruce.

Frank

 

[microstrip]

The Problem with A-B'ing and Why Neil Young is Right about Sound Quality.
by Allen Farmelo on April 4, 2012 - Comments (26)
(...)

Bruce,

Thanks for this great post by Allen - I agree with most of the views presented. However I disagree with the "good coffee" analogy. May be coffee drinking is an acquired taste, but music enjoying is not and will not need training. Some people will tell the opposite - training can sometimes modify your perception of sound quality and musical enjoyment.

 

[NorthStar]

But he did say; "I think the same thing can be said for the resolution of music, ..."

Methinks that his analogy, from his thinking perspective, is quite valid.

 

[rbbert]

OK, I am an odd man out here. In my experience, not days or hours but minutes or seconds, switching between tracks, allowed me to hear any differences. And hearing differences did not mean accuracy as much as hey, I like the sound of this over that.

For me to discern accuracy, I would have to hear the live unamplified instrument then the recording and go back and forth (but hey, damn sure would recognise the live recording everytime, so just stuck with comparing recordings, which takes away the live reference, and since my hearing memory is probably in the seconds range, its back to what "sounds" best, if there is any audible diffrence at all.

Tom

Are you talking about comparing 16/44.1 music with higher-res digital formats (24/44.1, 24/96, DSD, etc)? That would be cool, because as noted elsewhere in this forum, few "objectivists" seem willing to admit to differences between CD and SACD (for example) in DBT's.

 

[Thomas.Dennehy]

You can bias an A/B test to reach the desired conclusion. Same with focus groups -- if you phrase the question(s) correctly, you can get a focus group to tell you anything you want to hear.

 

[DonH50]

What questions? OK, that sounds argumentative, but actually I really am curious what questions are asked during or after an AB (or ABX) test?

I have participated in a few, but IME the only question was "A or B?" There were often some introductory remarks that certainly could bias folk, like "listen for warmth vs. coldness in the midrange" and other leading comments. However, for AB(X) testing in particular, there weren't any real "questions". That said, we did have some interesting discussions after the results were turned in about what people heard, "did you hear this or that in A or B" sort of things. It was interesting to (try to) correlate the post-discussion with the actual results. Interesting, often humbling, and at times annoying to various listeners. It also often led to refining the tests to focus on discussion points.

 

[garylkoh]

What questions? OK, that sounds argumentative, but actually I really am curious what questions are asked during or after an AB (or ABX) test?

I have participated in a few, but IME the only question was "A or B?" There were often some introductory remarks that certainly could bias folk, like "listen for warmth vs. coldness in the midrange" and other leading comments. However, for AB(X) testing in particular, there weren't any real "questions". That said, we did have some interesting discussions after the results were turned in about what people heard, "did you hear this or that in A or B" sort of things. It was interesting to (try to) correlate the post-discussion with the actual results. Interesting, often humbling, and at times annoying to various listeners. It also often led to refining the tests to focus on discussion points.

Exactly right - when we did the ABX testing with vinyl and digitized vinyl, the comments made were often more illuminating than the actual ABX results would have suggested. After that test, I further modified my DAC, and got the results even closer..... What we learn from not learning from ABX is more than the learning we would learned from ABX testing.

 

[fas42]

What we learn from not learning from ABX is more than the learning we would learned from ABX testing.

Huuuhh?! :b

Frank

 

[DonH50]

A negative result can be more useful than a positive in some cases. The post-test discussions often open new avenues of research. A positive result sometimes tells you nothing more than what you already knew.

 

[fas42]

Nicely elaborated, Don. I was starting to choke on all those "learn"s ... :b

Frank

 

[garylkoh]

A negative result can be more useful than a positive in some cases. The post-test discussions often open new avenues of research. A positive result sometimes tells you nothing more than what you already knew.

Thanks, Don. That's far clearer than my gobbledygook.

 

[DonH50]

Said the same thng, I just used more words.