Double Blind tests *did* show amplifiers to sound different

[dingus]

Why are you arguing with me? I'm the last person to defend blind testing and I've posted my comments about its issues on the board many times.

(-snip-)

You wrote as if you weren't up on Krueger and crew's ABX work and all I was trying to do is provide some background.

So there's really no need to lecture me about ABX testing.

sorry, wasnt meaning to lecture or to argue against anything you said, and from the sounds of it we dont disagree much on ABX. i just didnt get what the issue (ABX and amps sounding different) was all about to begin with.

 

[mep]

What do you think we can learn from the Pepsi challenge?

That coke tastes better?

 

[Whatmore]

That coke tastes better?

 

[MylesBAstor]

It's not hard to understand. You swear you hear an obvious difference. You trust your ears. But when you remove the visual reference, the difference is not only not obvious, it is difficult (or impossible) for you to discern which is which. Do you still trust your ears or do you need a controlled, scientific study? Been there done that. It was humbling. And it made me into a very different kind of audiophile. But I'm still not one who believes all amplifiers sound the same.

Tim

And what about blind audiophiles? I even know a very well respected, sight challenged person that owns an audio store in NJ. Where's their bias? Perhaps Mr. Olive should have been creative and selected a blind audience instead of using curtains and one speaker in the middle of a room.

 

[MylesBAstor]

If amps share similar topologies or can be made to null to a certain depth, then certified golden ears can not tell the difference between them (fact stereophile test years ago aka carver challenge) , and if one is $700 and the other $17,000 then what is the point of the higher priced unit.........well, its pride of ownership of course....

That of course JA has talked about ad naseum--not to mention the whole story was never printed -- and been discredited. But why let facts get in the way of a good story?

 

[Phelonious Ponk]

Tim,

No, sighted listening is not casual listening if you know how to listen and analyze it. BTW, the question is not that the blind tests are non-scientific, it is that IMHO all the humbling cases I have seen described by WBF members are flawed, either by statistics or method.

Of course sighted listening, no matter how seriously you might take it, is by definition flawed, as it introduces all kinds of potentially predudicial influences beyond what you are supposed to be evaluating--sound. And, of course, you can listen and analyze with exactly the same know-how, blind or sighted. I don't see anything in your answer that explains why you trust your ears to faithfully evaluate sighted listening with just a bit of self-made know-how and analysis, yet demand rigorous scientific methodology and statistical significance, "proof" even, when you don't know what gear is being played.

Tim

 

[MylesBAstor]

Of course sighted listening, no matter how seriously you might take it, is by definition flawed, as it introduces all kinds of potentially predudicial influences beyond what you are supposed to be evaluating--sound. And, of course, you can listen and analyze with exactly the same know-how, blind or sighted. I don't see anything in your answer that explains why you trust your ears to faithfully evaluate sighted listening with just a bit of self-made know-how and analysis, yet demand rigorous scientific methodology and statistical significance, "proof" even, when you don't know what gear is being played.

Tim

So if we love everything we look at, and the more expensive the better, then how do you explain that we don't like 80% of the stuff we listen to?

 

[Phelonious Ponk]

And what about blind audiophiles? I even know a very well respected, sight challenged person that owns an audio store in NJ. Where's their bias? Perhaps Mr. Olive should have been creative and selected a blind audience instead of using curtains and one speaker in the middle of a room.

You do understand, I hope, that "blind" in this context is a metaphor, that it means a lack of knowledge of what brands and models of equipment are being played when trying to evaluate sound, not brand, reputation, etc? And you also understand that an audiophile or shop owner could be phisically blind and still engage in "sighted" listening?

Tim

 

[Phelonious Ponk]

So if we love everything we look at, and the more expensive the better, then how do you explain that we don't like 80% of the stuff we listen to?

I don't think you love everything you look at, or that you necessarily believe "the more expensive the better." Neither of those beliefs are a part of this conversation, as far as I can tell.

Tim

 

[MylesBAstor]

You do understand, I hope, that "blind" in this context is a metaphor, that it means a lack of knowledge of what brands and models of equipment are being played when trying to evaluate sound, not brand, reputation, etc? And you also understand that an audiophile or shop owner could be phisically blind and still engage in "sighted" listening?

Tim

You do understand, I hope, that blind = blind. What part of that don't you understand? It doesn't make any difference whether or not he or she knows the brands; they don't actually know what they're listening to unless told and even then you could lie.

 

[MylesBAstor]

I don't think you love everything you look at, or that you necessarily believe "the more expensive the better." Neither of those beliefs are a part of this conversation, as far as I can tell.

Tim

Oh all of a sudden anticipation bias doesn't exist?

 

[Whatmore]

If someone wants to shut their eyes when auditioning equipment, if they deliberately don't want to know which component thay are listening to, what is the problem? What possible disadvantage does this put them at ?

 

[TJE]

I for one find that it is valuable to know what is or is not audible, using only your ears. People have experiences which are perceived as sound, but which can't be shown to start at the eardrums, either from looking at the sound itself (as sampled by a microphone and analyzed with software) or from listening alone (without sight). I think folks react negatively to the terms "blind" and "test", and a better phrase would be "pure listening evaluation". All it takes is a tablecloth over the equipment rack, so you can't see what's back there, and nobody has to be blinded or tested, you just listen. If the result is not as expected, then that's something to think about (and doesn't prove that the test is invalid).

 

[Orb]

From ITU BS.1116-1 "METHODS FOR THE SUBJECTIVE ASSESSMENT OF SMALL IMPAIRMENTS IN AUDIO SYSTEMS INCLUDING MULTICHANNEL SOUND SYSTEMS"

6. Programme material
Only critical material is to be used in order to reveal differences among systems under test. Critical material is that which
stresses the systems under test. There is no universally “suitable” programme material that can be used to assess all
systems under all conditions. Accordingly, critical programme material must be sought explicitly for each system to be
tested in each experiment. The search for good material is usually time-consuming; however, unless truly critical
material is found for each system, experiments will fail to reveal differences among systems and will be inconclusive.
It must be empirically and statistically shown that any failure to find differences among systems is not due to
experimental insensitivity because of poor choices of audio material, or any other weak aspects of the experiment, before
a “null” finding can be accepted as valid. In the extreme case where several or all systems are found to be fully
transparent, then it may be necessary to program special trials with low or medium anchors for the explicit purpose of
examining subject expertise (see Appendix 1).
These anchors must be known, (e.g. from previous research), to be detectable to expert listeners but not to inexpert
listeners. These anchors are introduced as test items to check not only for listener expertise but also for the sensitivity of
all other aspects of the experimental situation.
If these anchors, either embedded unpredictably within the context of apparently transparent items or else in a separate
test, are correctly identified by all listeners in a standard test method (§ 3 of this Annex) by applying the statistical
considerations outlined in Appendix 1, this may be used as evidence that the listener’s expertise was acceptable and that
there were no sensitivity problems in other aspects of the experimental situation. In this case, then, findings of apparent
transparency by these listeners is evidence for “true transparency”, for items or systems where those listeners cannot
differentiate coded from uncoded versions.

Yep and we both know how often the audio ABXers follow this precisely
Not often because I have never seen an online discussion of audio ABX ever utilising what I mentioned or the ITU standard on chosen material, and why those deciding to do ABX should take a long time in selecting/"engineering" content for traits-characteristics as I touched upon before rather than playing a whole or most of a music track looking for timbre-tone-attack/decay of the music in general/etc.
Cheers
Orb

 

[FrantzM]

Hi

I find amusing that the discussion is turning into a criticism of DBT. No one claims DBT is perfect, but in this world of shading and layers... It is the best methodology we know of. If we were to that this question: Is it sighted testing a more reliable methodology? The answer is a resounding NO!! And no amount of intellectual gymnastics , rejection of science, incantation of the magic will change this fact.

This said, I am not in the camp who claims that all amps sound the same. And to me this is perfectly reasonable and indeed DBT did show that is the case. This is the OP. We need more of those DBT to ascertain what is to me at least, a fact: Electronics do sound different on most speakers. I simply can't undrstand why many are opposed to the removal of knowledge which is what the "blind" stands for in the methodology not the actual blinding of a person...

 

[microstrip]

Sighted listening and non-scientific unsighted listening are exactly the same thing: Casual listening. And neither proves anything. But if you readily accept what you hear with full knowledge of every component and cable in the system (trust your ears), why do you require scientific vigor to accept what you hear without that knowledge? Quite the double standard. And a very weak case for trusting your ears.

Tim

Tim,

As you say, both methods have limitations and problems. I (and many others much better than me in long articles) have tried to explain how we deal with those of blind listening in high-end and had success using it, as proved by the excellence of many developments and systems that are not a consequence of the hazard. Unfortunately most of partisans of blind tests know pretty well how to bad-mouth sighted listening, an activity where they spend perhaps 95% of their anti high-end time, but give us very little about blind tests, with the exception of references to MP3 algorithms and the Harman loudspeaker tests.

Even in the so called objective forums we find almost no details about real blind tests or systems assembled using blind tests. Why this lack of enthusiasm and sharing?

 

[microstrip]

(...) If we were to that this question: Is it sighted testing a more reliable methodology? The answer is a resounding NO!! And no amount of intellectual gymnastics , rejection of science, incantation of the magic will change this fact.
(...)
Electronics do sound different on most speakers. I simply can't undrstand why many are opposed to the removal of knowledge which is what the "blind" stands for in the methodology not the actual blinding of a person...

Frantz,

We can dream about the best and more reliable methodology for a task, but if it can not be carried with the available resources and in useful time it becomes useless. IMHO the problem of blind testing is not the concept, it is that in order to carry them in a practical way people carry them in too particular conditions that do not allow any generalization of what was tested, and 99% of the times poor statistical analysis.

Again, what tests did you participate (or do you know about with complete details) carried in conditions of "removal of knowledge" - I agree it is much nicer than "blind", but I admit that any WBF member should know about the meaning of this word in audiophile jargon .

 

[FrantzM]

microstrip

Blind testing is not a dream. it is more reliable and repeatable than sighted tests. it can be carried by almost anyone. You remove the knowledge of the DUT, a simple protocol. Again no one claims perfection. As a practical matter I perform my evaluation of equipment sighted. i know which amp I am listening to and I try to remove my bia. i think I fail consistently on removing the biases .. I am back to a Burmester 911 amplifier ..I am patiently waiting for a 077 preamp to fall into my price trap .. I can tus appreciate the power of our belief and wants and likes... aka biases, they are strong and corrupt our evaluation. I would think that a reviewer could use the best tool available, so far few if any do that... That is the way of the High End AUudio market.

As for tests I participated in , mine were documented here in WBF and maybe AVS Forum. Cables knowledge were removed from me I could not see which speaker cables were used 6 AWG vs some expensive Audiophile brands. I wasn't able to recognize my cable versus the 6 AWG. other with better system than I here on the WBF who performed similar experiment didn't fare any better. Under similar conditions however, I was able to recognize my amplifier. SPL was matched at < 1 dB with test tones a 1 KHz. It didn't take me long to recognize my amplifier the 911 when compared to two very good amplifier (Mark Levison) and another non-US amplifer whose brand escape me could have been an Electrocompaniet or a Perreaux ... The test took a long time I will grant you that but it wasn;t the logistics nightmare you tend to portray it as.

One must admit at the end that sighted protocol are less reliable than unsigted ones. This is beyond an opinion, it is a fact.

 

[Orb]

Frantz,
what methodology did you use to select the music/sounds in your testing and what was the purpose and scope of what you hoped to achieve?
This is part of the challenge that Micro and myself have been highlighting (not the only one), which is further emphasised by the ITU quote Micro provided and my points earlier.
Cheers
Orb

 

[FrantzM]

Orb

I had heard about a similar test performed on the AVS Forum. To me it would be clear that the audiophile cable would be easily distinguishable from the other cable. I needed to make sure for myself. Prior to that I was of the opinion that blind tests were full of air. In sighted tests I had no problem whatsoever distinguishing things ...
I went with music and cuts I knew very well for having played them ad infinitum mostly Jazz and Western Classical.. To this day some have remained my goto. Medium was CD. That was at least 8 years ago.
I and some other people listened to the cuts and wrote opinion about the differences we heard. short cuts, about 1~2 mins, we could ask to replay. What was amusing to me where the inference of superiority, in two instances I thought I was listening to the audiophile cables when in fact it was the 6 AWG or the contrary I was listening to the audiophile cable but wrote i was the 6 AWG, in fact I could hear no difference for the most part. The substitution were quasi-random. i would leave the (my) room and come back to listen... about 10 tests. I did tire after 7 tests .. After the tests I knew that the 6 AWG were in 4 trials and the audiophile cable in 6. It could be sheer (un)luck that I failed in most of the tests but it told me how much our biases color our perception. Previously I was certain that I would pick my favorite cable easily since I had compared it before with other highly touted audiophile cables ...it was not so in this (blind) experience. To me that was enough. For some it may not be. No harm done. Difference of strokes

Some months after I and other audiophile were utterly confused by needle drops. We thought it was an LP playing when it was the CD, arm was lowered and music started playing .. We all thought it was an LP (including some heavy analog fans mostly anti-digital) .. All of us except the person playing the (cruel) joke. Of course once it was known that these were needle drops all the "wider soundstage", the "density", "naturalness" and my new favorite "organic" as opposed to chemical fertilizer grown ..in hindsight... I have since moved toward the "objectivist" side, especially about cables and admit to be skeptical of some extraordinary claims from some of my fellow audiophiles manage to make : things like different sound when cables have moved ... or the quasi-religious mantra of :"Trust your ears".

To repeat I do not think all amps sound the same, nor DAC, amps or preamps...Speakers differences are OTOH gross enough no BT needed but there again it could be helpful. and I believe that blind testing is a better tool than sighted. I repeat however that my evaluation of components are sighted. I would have preferred some reviewers to subject themselves to blind from time to time but know it is difficult but not impossible. I also believe that some audiophile claims can be sustained by BT. Why not use a better tool whenever possible? To me it simply make sense.. That is all. Let's not take the debate to the absurd.