Double Blind tests *did* show amplifiers to sound different

[microstrip]

microstrip

Blind testing is not a dream. it is more reliable and repeatable than sighted tests. it can be carried by almost anyone. You remove the knowledge of the DUT, a simple protocol. Again no one claims perfection. As a practical matter I perform my evaluation of equipment sighted. i know which amp I am listening to and I try to remove my bia. i think I fail consistently on removing the biases .. I am back to a Burmester 911 amplifier ..I am patiently waiting for a 077 preamp to fall into my price trap .. I can tus appreciate the power of our belief and wants and likes... aka biases, they are strong and corrupt our evaluation. I would think that a reviewer could use the best tool available, so far few if any do that... That is the way of the High End AUudio market.

As for tests I participated in , mine were documented here in WBF and maybe AVS Forum. Cables knowledge were removed from me I could not see which speaker cables were used 6 AWG vs some expensive Audiophile brands. I wasn't able to recognize my cable versus the 6 AWG. other with better system than I here on the WBF who performed similar experiment didn't fare any better. Under similar conditions however, I was able to recognize my amplifier. SPL was matched at < 1 dB with test tones a 1 KHz. It didn't take me long to recognize my amplifier the 911 when compared to two very good amplifier (Mark Levison) and another non-US amplifer whose brand escape me could have been an Electrocompaniet or a Perreaux ... The test took a long time I will grant you that but it wasn;t the logistics nightmare you tend to portray it as.

One must admit at the end that sighted protocol are less reliable than unsigted ones. This is beyond an opinion, it is a fact.

Frantz,

A full well lived audiophile lifetime, and just two poorly documented experiences to report ... BTW, 1dB matching allows easy identification ...

And yes, I am adressing the listening tests to optimize stereo systems, not to carry wire bets. And I often use independent non audiophile type listeners to control my own opinions and get statistics.

Properly carried blind tests are surely more reliable, but poorly carried they are misleading and do not allow any generalization.

 

[KeithR]

and we know that its been demonstrated that ss can sound indistinguishable from tubes, and yes different speakers will reveal differences in amps as well.

I don't believe that has been demonstrated. I've never heard SS that is indistinguishable from tubes either. Then again, if you think "tube sound" is tilted up midrange and tubby bass, that's not what I'm talking about.

 

[dingus]

I don't believe that has been demonstrated. I've never heard SS that is indistinguishable from tubes either. Then again, if you think "tube sound" is tilted up midrange and tubby bass, that's not what I'm talking about.

this is in reference to the Carver demo...

 

[Whatmore]

Frantz,

A full well lived audiophile lifetime, and just two poorly documented experiences to report ... BTW, 1dB matching allows easy identification ...

And yes, I am adressing the listening tests to optimize stereo systems, not to carry wire bets. And I often use independent non audiophile type listeners to control my own opinions and get statistics.

Properly carried blind tests are surely more reliable, but poorly carried they are misleading and do not allow any generalization.

How can they be any more misleading than sighted tests?

 

[microstrip]

How can they be any more misleading than sighted tests?

Because we miss information and knowledge of how to carry proper blind tests and how analyze them. In a sighted test you know you have to deal with the expectation bias, well documented and easily understood. In a blind test, people that seem to ignore how hard it is to carry a proper blind test and the absolute necessity of carrying positive control tests will persuade you that since the pseudo-test was blind it becomes the "humbling truth".

Did you notice that there are no real debates on blind test methodology and analysis in audio forums? I would suppose that the enthusiasts of blind tests would love to share their knowledge and experience.

 

[Whatmore]

Because we miss information and knowledge of how to carry proper blind tests and how analyze them. In a sighted test you know you have to deal with the expectation bias, well documented and easily understood. In a blind test, people that seem to ignore how hard it is to carry a proper blind test and the absolute necessity of carrying positive control tests will persuade you that since the pseudo-test was blind it becomes the "humbling truth".

Did you notice that there are no real debates on blind test methodology and analysis in audio forums? I would suppose that the enthusiasts of blind tests would love to share their knowledge and experience.

Lets compare two scenarios here:
1) Conduct a "casual" sighted test with all the known compromises: expectation bias, rough level matching (if at all), knowledge of which component is playing - to name a few. ..... You hear a clear difference between the two components

2) Without changing a thing in the setup, conduct the same test with your eyes shut. ..... The difference disappears to the extent that you can no longer reliably pick which component is which

Do we conclude that
a) Despite our knowledge of our own biases, it is clear that we hadn't fully eliminated them in the sighted test and whilst flawed, the blind test did manage to eliminate enough bias that we could no longer tell the components apart

or
b) We ignore the results of the blind test and decide that we had in fact completely overcome our biases in the sighted test and that there really are audible differences

 

[Phelonious Ponk]

You do understand, I hope, that blind = blind. What part of that don't you understand? It doesn't make any difference whether or not he or she knows the brands; they don't actually know what they're listening to unless told and even then you could lie.

No, I don't understand that. In this context blind means without knowledge. All he has to do is have knowledge of what he's listening to through any other means than listening alone, and the listening is not "blind."

Tim

 

[Phelonious Ponk]

Oh all of a sudden anticipation bias doesn't exist?

Of course it exists. Do you think it is impossible to come to a listening session with a prior negative impression of something that is expensive or attractive?

Tim

 

[Phelonious Ponk]

Tim,

As you say, both methods have limitations and problems. I (and many others much better than me in long articles) have tried to explain how we deal with those of blind listening in high-end and had success using it, as proved by the excellence of many developments and systems that are not a consequence of the hazard. Unfortunately most of partisans of blind tests know pretty well how to bad-mouth sighted listening, an activity where they spend perhaps 95% of their anti high-end time, but give us very little about blind tests, with the exception of references to MP3 algorithms and the Harman loudspeaker tests.

Even in the so called objective forums we find almost no details about real blind tests or systems assembled using blind tests. Why this lack of enthusiasm and sharing?

You completely avoided the point, but I'll answer you anyway. I'm not badmouthing sighted listening. It's the only kind I've done in a long time, because I know what's in my system. But if you're listening (key word here) for differences between components, media, whatever, removing the knowledge of what component is playing when, removes the ability of that knowledge to impact the results. And while you revere and respect your listening sessions with all of that blatant potential for bias left deliberately in, you demand scientific perfection from the purer audio experience of listening blind, for fear of the subtle, unknown, possible bias?

Methinks thou doth protest too much.

Tim

 

[Phelonious Ponk]

Because we miss information and knowledge of how to carry proper blind tests and how analyze them. In a sighted test you know you have to deal with the expectation bias, well documented and easily understood. In a blind test, people that seem to ignore how hard it is to carry a proper blind test and the absolute necessity of carrying positive control tests will persuade you that since the pseudo-test was blind it becomes the "humbling truth".

Did you notice that there are no real debates on blind test methodology and analysis in audio forums? I would suppose that the enthusiasts of blind tests would love to share their knowledge and experience.

Let's forget about "tests" for a moment. I've never run a "test" in a system, at home or elsewhere. But I've compared lots of gear, in my system and in others. Let's say you're comparing two digital cables between player and DAC. You are very familiar with the brands, the prices, the reviews, the construction of the two cables, but you don't care about any of that, you just want to compare the sound, so you have a friend set up the connection and you listen very carefully to a revealing passage in music you know well. Then he switches the cables, and repeats the process several times. Forget A & B. Sounds too much like a test. You just tell your friend which one you like better in each pair listened to.

Now, alternately, your friend tells you, every time, which cable you're listening to before you listen. What have you gained? How can this sighted listening lead to any different conclusions? How can this be a higher level of "trusting your ears?"

Tim

 

[FrantzM]

Lets compare two scenarios here:
1) Conduct a "casual" sighted test with all the known compromises: expectation bias, rough level matching (if at all), knowledge of which component is playing - to name a few. ..... You hear a clear difference between the two components

2) Without changing a thing in the setup, conduct the same test with your eyes shut. ..... The difference disappears to the extent that you can no longer reliably pick which component is which

Do we conclude that
a) Despite our knowledge of our own biases, it is clear that we hadn't fully eliminated them in the sighted test and whilst flawed, the blind test did manage to eliminate enough bias that we could no longer tell the components apart

or
b) We ignore the results of the blind test and decide that we had in fact completely overcome our biases in the sighted test and that there really are audible differences

b) is the norm.

 

[microstrip]

Lets compare two scenarios here:
1) Conduct a "casual" sighted test with all the known compromises: expectation bias, rough level matching (if at all), knowledge of which component is playing - to name a few. ..... You hear a clear difference between the two components

2) Without changing a thing in the setup, conduct the same test with your eyes shut. ..... The difference disappears to the extent that you can no longer reliably pick which component is which

Do we conclude that
a) Despite our knowledge of our own biases, it is clear that we hadn't fully eliminated them in the sighted test and whilst flawed, the blind test did manage to eliminate enough bias that we could no longer tell the components apart

or
b) We ignore the results of the blind test and decide that we had in fact completely overcome our biases in the sighted test and that there really are audible differences

Any conclusion you take from imaginary scenarios described in two lines is meaningless. The main question is this discussion is that the practical restrictions imposed by casual blind tests obfuscate the existing differences. IMHO it is the true reason why you will only find very few detailed descriptions of valid blind tests in the net, and people always say that THEIR own listening is sighted, but OTHERS should listen blind.

BTW, since I (and many others) know that 99.99% of the amateur CASUAL blind tests are carried in conditions that will falsify the result I choose b)

 

[Whatmore]

Any conclusion you take from imaginary scenarios described in two lines is meaningless. The main question is this discussion is that the practical restrictions imposed by casual blind tests obfuscate the existing differences. IMHO it is the true reason why you will only find very few detailed descriptions of valid blind tests in the net, and people always say that THEIR own listening is sighted, but OTHERS should listen blind.

BTW, since I (and many others) know that 99.99% of the amateur CASUAL blind tests are carried in conditions that will falsify the result I choose b)

OK if you don't agree with my conclusions as being in any way valid then what conclusions would you draw from the two scenarios?

EDIT: BTW what exactly are the "practical restrictions imposed by casual blind tests" ?

 

[microstrip]

b) is the norm.

Frantz,

Should we think the great majority of the others are wrong and only a few carry the truth? That all the audio knowledge about the so called small differences assembled using during decades is intrinsically false?

BTW, do you believe that the final critical listening tests of Burmester (or Audio Research) electronic equipment before shipping are carried in blind conditions?

 

[microstrip]

Let's forget about "tests" for a moment. I've never run a "test" in a system, at home or elsewhere. But I've compared lots of gear, in my system and in others. Let's say you're comparing two digital cables between player and DAC. You are very familiar with the brands, the prices, the reviews, the construction of the two cables, but you don't care about any of that, you just want to compare the sound, so you have a friend set up the connection and you listen very carefully to a revealing passage in music you know well. Then he switches the cables, and repeats the process several times. Forget A & B. Sounds too much like a test. You just tell your friend which one you like better in each pair listened to.

Now, alternately, your friend tells you, every time, which cable you're listening to before you listen. What have you gained? How can this sighted listening lead to any different conclusions? How can this be a higher level of "trusting your ears?"

Tim

Tim,

Thanks for making my answer simple - comparing cables takes a long time and proper listening. The ridiculous casual test you describe will result surely in a meaningless result, either blind or sighted.

 

[microstrip]

OK if you don't agree with my conclusions as being in any way valid then what conclusions would you draw from the two scenarios?

EDIT: BTW what exactly are the "practical restrictions imposed by casual blind tests" ?

Sorry but the proper answer will takes a long time. Number of tests, time between them, settling time of equipment, listener fatigue, precedence effect, statistical analysis and the need for positive control tests are just a few. Read about how blind tests are carried by Harman and others and you will immediately see the difference. BTW, their tests were optimized for speaker development, considering the "small differences" due to source, electronics and cables can be ignored.

 

[Whatmore]

Sorry but the proper answer will takes a long time. Number of tests, time between them, settling time of equipment, listener fatigue, precedence effect, statistical analysis and the need for positive control tests are just a few. Read about how blind tests are carried by Harman and others and you will immediately see the difference. BTW, their tests were optimized for speaker development, considering the "small differences" due to source, electronics and cables can be ignored.

I think perhaps we are each talking about completely different things here.

 

[Phelonious Ponk]

Any conclusion you take from imaginary scenarios described in two lines is meaningless. The main question is this discussion is that the practical restrictions imposed by casual blind tests obfuscate the existing differences. IMHO it is the true reason why you will only find very few detailed descriptions of valid blind tests in the net, and people always say that THEIR own listening is sighted, but OTHERS should listen blind.

BTW, since I (and many others) know that 99.99% of the amateur CASUAL blind tests are carried in conditions that will falsify the result I choose b)

And what are the practical restrictions of having the components being compared switched out of the listener's site?

Tim

 

[Phelonious Ponk]

Tim,

Thanks for making my answer simple - comparing cables takes a long time and proper listening. The ridiculous casual test you describe will result surely in a meaningless result, either blind or sighted.

We've had people here report astonishing differences between cables at first listen. But I'm sure it takes weeks of sighted listening and discussion to hear subtle differences and reach audiophile consensus.

Tim

 

[microstrip]

I think perhaps we are each talking about completely different things here.

I am always addressing the same thing - the constraints imposed by real audiophile life added to our ignorance about audio blind tests methodology make the so called casual tests invalid. So the only thing left for forum debates is gedankenexperiment.

BTW, did you ever carry a blind test you consider valid?