Double Blind Testing and the threshold of necessity

[RUR]

Blind testing can be as inaccurate as sighted testing if it's done poorly enoug(h),

My point, exactly, Tim.

though it takes a bit of effort to bias it as much as sighted testing can manage with no effort at all.

Of course bias isn't the issue with flawed DBT's. When they're flawed, it's for other fundamental reasons, to which sighted tests are equally susceptible. Sighted or double blind, a sufficiently flawed test has no value, not marginal value. To put it differently: There's no preference for a fundmentally flawed DBT over an intrinsically flawed sighted test, and it's unfortunate that folks are citing such flawed DBT's as evidence of one thing or another.

Again, you're preaching to the choir about the virtues of properly conducted blind (vs. sighted) tests, in general.

 

[Gregadd]

Not that you guys don't already know this but it bears repeating. Blind testing does not so much mean eyes closed as it it does eliminating variables. Then the result is based on the factor under consideration rather than some extraneous factor. Obviously one could not pick the best the with your eyes closed.

 

[Phelonious Ponk]

Sighted or double blind, a sufficiently flawed test has no value, not marginal value. To put it differently: There's no preference for a fundmentally flawed DBT over an intrinsically flawed sighted test, and it's unfortunate that folks are citing such flawed DBT's as evidence of one thing or another.

I'm not sure I'm getting my point across, though I've made it more than once. Does a badly flawed DBT have no statistical value? None. I agree. Where we may disagree is on the value of flawed, informal, statistically invalid blind listening. I think it has value as it is the only way, in my opinion, that listening can be critical. When a group of guys get together to compare preamps (or whatever) with their eyes wide open and their prejudices fully engaged, it's a listening party, nothing more. Enjoy, but it's a waste of time if you're looking to evaluate anything critically. Close your eyes and you take a huge step forward. Statistically valid? Nope. But way ahead of the alternative.

Tim

 

[RUR]

I'm not sure I'm getting my point across, though I've made it more than once. Does a badly flawed DBT have no statistical value? None. I agree. Where we may disagree is on the value of flawed, informal, statistically invalid blind listening. I think it has value as it is the only way, in my opinion, that listening can be critical. When a group of guys get together to compare preamps (or whatever) with their eyes wide open and their prejudices fully engaged, it's a listening party, nothing more. Enjoy, but it's a waste of time if you're looking to evaluate anything critically. Close your eyes and you take a huge step forward. Statistically valid? Nope. But way ahead of the alternative.

Tim

But Tim, listening can't be "critical", at least insofar as I'd define it, if the statistics and/or test conditions are flawed. It's simply another form of listening party, except with a high geek-factor. I'm afraid I'll have to disagree that such sessions have any more value than a sighted version.

 

[Phelonious Ponk]

But Tim, listening can't be "critical", at least insofar as I'd define it, if the statistics and/or test conditions are flawed. It's simply another form of listening party, except with a high geek-factor. I'm afraid I'll have to disagree that such sessions have any more value than a sighted version.

We'll have to agree to disagree then, RUR. I believe anything that gets around expectation bias increases the value of the listening and the assessment. And if I don't know which component I'm listening to, I've bi-passed my expectations of the components in question. Could I still hear a difference because I expected to hear a difference? Or not? Sure. But I often heard it backwards, which is telling. I absolutely believe that I greatly reduce the incidence of bias by listening blind. I've done it many times at home. And many times I've found that difference I thought I heard disappeared when I could no longer see which component.player/file I was listening to. It works for me. I would not claim it as objective proof, but I would, and have said I had been listening to X and Y for two weeks and was absolutely convinced that Y revealed more detail and had a broader image, but when, at the end of the two weeks, I set it up and switched back and forth between them blind, I could no longer tell which was which. I sent Y back, got a refund, and am content that component X is every bit as good as Y in my system.

YM, of course, MV.

Tim

 

[RUR]

Tim, it is you who are missing my point, but this is tiresome and I'm moving on.....

 

[Phelonious Ponk]

Tim, it is you who are missing my point, but this is tiresome and I'm moving on.....

Evidently I am. Enjoy the music.

Tim

 

[JackD201]

Speaking of parties....

Q: How do you know you are at an audiophile party?

A: You'll never hear songs played from start to finish.

 

[Randy Bessinger]

And anecdotal reports are ? "I heard this difference" "Difference that anyone could hear" etc. etc.

 

[Randy Bessinger]

Speaking of parties....

Q: How do you know you are at an audiophile party?

A: You'll never hear songs played from start to finish.

Funny!

 

[microstrip]

A: You'll never hear songs played from start to finish.

This raises another question : how long should last the tests in a DBT?

 

[Gregadd]

This raises another question : how long should last the tests in a DBT?

As long as it takes for form a lasting impression. How would you assess things like PR&T, listening fatigue, Spatial dimensions,bass response in a short test with rapid switching.

 

[Ethan Winer]

Has anyone ever done any studies on whether our eyes help us hear?

If mep hadn't already linked to the McGurk effect I would have because it's a great example. Aside from that, I think seeing helps in other ways. Last night I visited Mark "basspig" Weiss to watch an orchestra concert he and I video'd two weeks ago. I've noticed this effect before, but last night really drove it home. I was watching and listening to some passage in the orchestra and didn't even notice the fairly prominent snare drum part being played. Then the camera highlighted the snare drum guy and all of a sudden it was easy to notice the snare audibly. It's well known that we can't listen to every detail all at the same time in a complex piece. So in this case the video helped to focus attention on what's sounding at that moment. I noticed this last night repeatedly throughout the performance (Scheherezade) as a soft flute part came into focus once the camera showed the player, and so forth.

--Ethan

 

[Gregadd]

Agreed Ethan. Partially deaf people find lip reading particulary helpful in their hearing.
Moreover the local deaf football team uses a bass drum for the snap count. (feeling the vibrations)

 

[RBFC]

Agreed Ethan. Partially DEAF people find lip reading particulary helpful in their hearing.
Moreover the local DEAF football team uses a bass drum for the snap COUNT. (feeling the vibrations)

I helped Gregadd out a bit, and bolded the corrections from his post above. Unless there's a new sport I'm unaware of, in which case I apologize.

Lee

 

[Gregadd]

Lee I think my way was funny. It's the glasses when I run the spell checker.

 

[RBFC]

Cool! I've got a new workout for the guys come next Saturday: Death Football. I'm so all over this! Designing my new helmet as we speak....

Lee

 

[Gregadd]

I got this from Bruce. How is this for listening?
[video]http://www.gearslutz.com/board/high-end/563413-speakers-too-detailed-i-can-hear-so-many-flaws.html[/video]

post #21

 

[JackD201]

Downright profound.