Comparative Listening Tests

[microstrip]

(...) If you say that any difference heard by a panel of audiophile listeners in sighted tests is real without any doubt and the panel fails to reliably hear this difference in a blind test, would you blame the blind test as such? (...)

You have a point, audiophile bragging and swashbuckling are the best friends of extreme objectivists ...

I never take seriously differences pointed by panels if members can see each other, much less if they can exchange opinions.

 

[microstrip]

What conditions exactly?

All those that you find in tests carried in audio forums by amateurs and pseudo-experts ...

What is a positive test?

A test that is carried with knowledgeable references that you use to assert and calibrate your own methodology and test. It should result in a positive identification of differences.

 

[KlausR.]

Originally Posted by KlausR.
"What conditions exactly?"

All those that you find in tests carried in audio forums by amateurs and pseudo-experts ...

That's too sweeping a statement to make any sense. What then would be a correctly set up blind test?

Originally Posted by KlausR.
"What is a positive test?"

A test that is carried with knowledgeable references that you use to assert and calibrate your own methodology and test. It should result in a positive identification of differences.

How do you determine that the differences are real and not imaginary?

 

[Gregadd]

I have said this before. Concealing the identity of the DUT is merely one aspect of a proper a/b test. Certain people over emphasize its importance for their own purpose. The real world is conducted with eyes wide open

 

[jkeny]

Listen, all this is deflection from a crucial point of KalusR's argument - even if you do find a difference, just turn up the volume like KlausR did & the difference will disappear so what's the point in all of this?

Oh & if you find a difference between speakers, no problem - just poke your finger through the midrange (or tweeter) of both of them & they will then sound exactly the same - see simple

Tests are for fools when you have a volume knob or a finger available - all audio devices sound the same - Klaus doesn't believe in them for himself - just for other fools!

 

[microstrip]

That's too sweeping a statement to make any sense. What then would be a correctly set up blind test?

We have debated it extensively in this forum - a correctly set up test under controlled conditions must result in positive identifications of known differences. A test resulting in a negative can have two causes - no difference or the test was not able to detect it.

Consider this example. Assume that it is accepted and proved that a certain SE amplifier sounds different from a SS amplifier. If in a particular test under blind conditions it is found that there is no difference, this implies that the test is not valid.

Just for fun, in what is considered very poorly controlled conditions, once I have tried changing remotely the treble level in my Soundlab speakers by 3dB and asking two friends to feel a scorecard in ABX methodology blind - no one got a positive identification. Surely this does not prove that we can not perceive this difference, just that the test was poorly carried.

How do you determine that the differences are real and not imaginary

Only analyzing the final outcome of the tests along time and cross checking results.

 

[KlausR.]

We have debated it extensively in this forum - a correctly set up test under controlled conditions must result in positive identifications of known differences. A test resulting in a negative can have two causes - no difference or the test was not able to detect it.

So far we are on the same line.

Assume that it is accepted and proved that a certain SE amplifier sounds different from a SS amplifier.

Ok. But proved by what listening method? Sighted? Blind?

If in a particular test under blind conditions it is found that there is no difference, this implies that the test is not valid.

Here again, we are on the same line.

 

[amirm]

I have said this before. Concealing the identity of the DUT is merely one aspect of a proper a/b test. Certain people over emphasize its importance for their own purpose. The real world is conducted with eyes wide open

It is. And unfortunately vast majority of outcomes are driven by those eyes, not ears alone.

The scientific community was there, and done that. And no longer allows any eyes involved in such evaluations. Same is true of medical world. Would you take medication that was tested with patients having full knowledge of it?

 

[amirm]

Like it or not live music is and must remain our standard. Otherwise we are just chasing our collective tail.

We are chasing our tail in audio. It is a completely broken standard where it is impossible to know what the talent heard that approved the sound. Just take the fact that different speakers were used than yours to approve said music. Clearly they heard a completely different sound than you. How on earth can you have any wish/dream let alone any entitlement to reality of what instruments sounded like?

Until we understand this important topic -- that we can never know what was there in the studio -- we will chase our tails. Once that is understood and accepted, then we can do some things about it.

That is the fatal flaw of the scientific/measurement approach. Perfect measures don't produce musical results.

Scientific method is about using ears and only ears to evaluate things. There is an entire field of psychoacoustics where everything in it is driven by listening tests. Measurements and system understanding combined with that gives us a total picture.

The random evaluation of audiophiles with their eyes open and brain full of prejudice and mistaken lay assumptions about how sound reproduction works, is a recipe for disaster.

Mind you, there is a benefit to subjective method of analysis. Anything that like to see work, magically does! It brings happiness that way. Science on the other hand, pours cold water on it. So I get why we don't want to accept what science says. This is a human hobby after all. What makes no sense is to keep insisting that there is correctness to this madness. There just isn't.

 

[jkeny]

It is. And unfortunately vast majority of outcomes are driven by those eyes, not ears alone.

The scientific community was there, and done that. And no longer allows any eyes involved in such evaluations. Same is true of medical world. Would you take medication that was tested with patients having full knowledge of it?

Are medicines 'tested' at home by hobbyists with scant knowledge of suitable procedures?
Please use comparisons that are rooted in some form of realistic scenario - yours is a very jaded & non-nonsensical point !

 

[amirm]

Welcome back amirm. I thought you had left to spend more time on the science forum. Do you care to respond to any of the posts that were directed to you?

What question and why?

I agree with Ron also, but perhaps not 100%. That is very absolute. I agree that we should continue to audition components, continue to pursue tweaks, and continue to listen for significant or subtle differences. Perhaps I could be more realistic, circumspect and skeptical about my and your expressed conclusions. We don't always attempt to do the best we can, but many of us do try. Intellectual honest is a bit tougher to judge in others, and in ourselves. I do realize that I fool myself all the time, about listening conclusions and many other observations.

Good. I look forward to this being reflected in your future posts.

Anyone else? Is it just Ron, Peter and I in this boat? Rest of you have super human hearing that cannot be mistaken as Ron explained?

 

[853guy]

It is. And unfortunately vast majority of outcomes are driven by those eyes, not ears alone.

The scientific community was there, and done that. And no longer allows any eyes involved in such evaluations. Same is true of medical world. Would you take medication that was tested with patients having full knowledge of it?

 

[cjfrbw]

My wife has super human hearing, I don't.

 

[amirm]

Are medicines 'tested' at home by hobbyists with scant knowledge of suitable procedures?

Hobbyist? No, what is objected to is the entire field of audio science as conducted for medical research and audio. That work has been conducted by major luminaries in the industry. Here is one of any examples: https://secure.aes.org/forum/pubs/conventions/?elib=6338

"AES CONVENTION PAPERS FORUM
Hearing is Believing vs. Believing is Hearing: Blind vs. Sighted Listening Tests, and Other Interesting Things"

The counter argument is indeed made by hobbyist who don't know the first thing about how music is produced, the science behind the equipment, psychoacoustics, etc. You rather put faith in that?

Please use comparisons that are rooted in some form of realistic scenario - yours is a very jaded & non-nonsensical point !

So you say. So you say....

 

[jkeny]

Hobbyist? No, what is objected to is the entire field of audio science as conducted for medical research and audio.

Where exactly are scientifically conducted blind tests being objected to here - specific links to posts please?

 

[DaveC]

Ok. But proved by what listening method? Sighted? Blind?

SS vs SET are going to have easily measurable differences in frequency response in most speakers due to the relatively high output impedance of the SET vs SS amp. We can prove differences via measuring sometimes.

 

[PeterA]

We are chasing our tail in audio. It is a completely broken standard where it is impossible to know what the talent heard that approved the sound. Just take the fact that different speakers were used than yours to approve said music. Clearly they heard a completely different sound than you. How on earth can you have any wish/dream let alone any entitlement to reality of what instruments sounded like?

Until we understand this important topic -- that we can never know what was there in the studio -- we will chase our tails. Once that is understood and accepted, then we can do some things about it.



Scientific method is about using ears and only ears to evaluate things. There is an entire field of psychoacoustics where everything in it is driven by listening tests. Measurements and system understanding combined with that gives us a total picture.

The random evaluation of audiophiles with their eyes open and brain full of prejudice and mistaken lay assumptions about how sound reproduction works, is a recipe for disaster.

Mind you, there is a benefit to subjective method of analysis. Anything that like to see work, magically does! It brings happiness that way. Science on the other hand, pours cold water on it. So I get why we don't want to accept what science says. This is a human hobby after all. What makes no sense is to keep insisting that there is correctness to this madness. There just isn't.

amirm, If what I highlighted in bold is true, how do you account for the success of systems like ddk's, MikeL.'s, and the system that 853guy just heard? Did they arrive at what they feel are successful systems by following a recipe for disaster? I think someone earlier in this thread observed that the method that most audiophiles use to select the gear in their systems does seem to work to the extent that many of us are very satisfied with how our systems sound.

Assuming that you use a different method, and perhaps even have different goals in this hobby, can I ask if you are satisfied with the sound of your system? You once promised to post details about your system, but I have not yet seen them. I did see a photograph of all of your testing gear in your room. Is that where you listen to music also?

 

[853guy]

What question and why?

Hi Amir,

How about the ones where I questioned your bias in posts #286 and #287? Missed those, huh?

Never mind, no biggie. After the many posts of yours I’ve read since joining the forum, I’m guessing - guessing, mind - your reply would more or less be something along the lines of:

—

Hi 853guy,

Thank you for your post.

Please allow me to deflect attention away from your points as if my background as Corporate VP at Microsoft obviates me from any responsibility to acknowledge them in light of my bias.

Here instead is a chart I’ve scanned from a textbook. Please note my use of red text, blue text and a different font when I’m doubling down on my bias. By the way, I used to work at Microsoft, did I tell you?

Attached is a photo of “Prof” Keith O. Johnson holding a Grammy, just like Milli Vanilli did.

I’ve also attached another chart of measurements I myself produced under sighted conditions, fully cognisant of the product’s name, its materiality, its build quality, its retail price and the fact that I am resentful I had to pony up to pay for it myself. No controls were necessary here because I decided specifically to subject this product to testing due to the fact if was from a manufacturer I do not sell through my high-end installation business.

Sean Olive.

Also, I’ve attached a stock image of something not related to your post, nor in fact, anything, really, followed by lots and lots of text about ABX tests I have “passed” and issue you a challenge to take them yourself, despite the fact they lack appropriate controls, are self-administered, have limited utility value and prove nothing except that on a particular occasion, a difference was detected to within a certain level of statistical confidence. But allow me to over-generalise on those results anyway.

A photo of my dog.

A lot of rhetoric in which I avoid the need to acknowledge any of the research that offers extremely robust, peer-reviewed, published, statistically-significant findings into experimenter bias, design bias, selection/sampling bias including omission bias and inclusivity bias, the halo effect, response bias, performance bias, procedural bias, measurement bias, reporting bias, citation bias, confounding, reactivity effect (1) and the John Henry Effect. None of those things apply to me anyway because I’ve won three Emmys.

Lots more text on the outrage that I need to justify my position here. Did I not already say I worked at Microsoft?

Floyd Toole.

There. Do you now see how much effort it is for me to deflect attention away from the inherent bias in many of my posts? If only you would take the ABX test you would appreciate that I use those tests very specifically, not because they have any statistical correlation to one’s ability to appreciate music via an intermediary electro-acoustic device, but in order to avoid any self-reflection on my part whether my agenda influences my ability to remain objective about anything I post. Also, I have lots and lots of practice detecting minute sonic differences, so I’m confident you won’t know what to listen for.

Oh. You took the ABX test and had results in which the level of confidence was greater than chance?

Let me give you another one I know I’ve already managed to “pass”.

BTW… link to my new forum: audiobiasreview.com

—

Did I more or less get the gist of them? Did I get close?

Amir, in all seriousness, you’ve had the chance to contribute something useful to this forum, and beyond that, potentially our pursuit of greater enjoyment from the gear we use to listen to music. The satirical tone above is deliberate and something I take full responsibility for, but in light of my earnest and genuine prior attempts to point out the bias that I and others believe corrupts your perspective, both in my replies to you on this forum and privately, I use this methodology as a last resort (2).

And despite being willing to learn something about an aspect of audio reproduction I’ve heretofore been naive in, your level of credibility - despite your many, many posts (or perhaps, because of them) - has declined steadily for me, and for the all the reasons related to the inherent bias I and others have mentioned previously in the posts you apparently can’t or won’t acknowledge.

“Scientific method is about using ears and only ears to evaluate things. (...) The random evaluation of audiophiles with their eyes open and brain full of prejudice (...) is a recipe for disaster.”

Do you know who said that? You did, in post #372. The exact same person who bought the “$300 AC outlet” to evaluate, and before the evaluation even began stated you were “confident (it) will do zero to improve audio performance”, while planning to conduct an uncontrolled test, sighted.

Using technical knowledge, or data, or graphs, or measurements, or ABX tests, or testing methodologies - without eliminating bias - and using them to further an agenda against those who do not share your world-view is not audio science. It’s audio zealotry. I cannot think of any term to more accurately describes your conduct here.

Of course, I could have simply turned a blind eye to your posts and ignored them as many already do, and perhaps, as you’ve done to me. But given your level of influence, not to mention the way in which you’ve conducted yourself on this forum and your refusal to acknowledge when you’ve been mistaken, intellectually dishonest or simply wrong, I feel compelled to respond because I see someone who has a lot to offer, if only your own agenda and bias could be put to one side. Me personally? I think that’s a shame.

I realise this is just my personal view on your conduct. I too have a bias, so I offer this counterpoint to you fully aware of its limits - it is after all, one person’s perspective. Whether you take it or leave it will be of no consequence to me, but may have continued consequences for the forum in which many good people have left because of your tactics. I know, because I’ve been contacted by some of them personally.

Nevertheless, I continue to wish you well in whatever it is you put your hand to. I hope that whatever that is, it will be built upon something more credible than the posts you make here.

My best to you,

853guy

(1) See the “Hawthorne Effect” in which subjects aware of being observed in a given study show greater variance and inconsistency of results, with both positive and negative skews. For more, see McCambridge, Witton & Elbourne (2014), https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3969247/

(2) For more on the use of satire as a critique of claims to power and authority in political discourse, see “The Impact of Real News about ‘Fake News’: Intertextual Processes and Political Satire”; Brewer, Goldthwaite Young & Morreale (2013); https://academic.oup.com/ijpor/arti...-News-about-Fake-News?redirectedFrom=fulltext

 

[jkeny]

And while you are answering 853guy's questions (LOL, as if this is going to happen) you can answer & own up to the false claims I have pointed out that you have made on this forum - USB re-transmission of packets, oversampling in FFTs to name just the most recent ones.

 

[microstrip]

So far we are on the same line.

Ok. But proved by what listening method? Sighted? Blind?

Here again, we are on the same line.

OK, make it blind, in a panel with more than 20 people in controlled conditions.

Do you see now why we need control tests resulting in positive identification?

And now I have answered your questions would you agree in answering mine that you failed to answer? Did you ever participate in any blind test involving audio electronics that you consider adequately carried and had an interesting outcome for audio development?