Admin note: this thread started at the end of jitter thread and good suggestion was made to make it its own thread. So here it is.
The power of blind testing comes from elimination of bias. It does that powerfully and can be abundantly easy to see and prove using real data. Its reverse role for finding small differences is much more difficult if not impossible to prove. To wit, I can make a change to the system that is measurable and strongly so, yet not found in a blind test. The fact that we cannot use objective data to determine if our objective tests is working puts us in a tough, tough situation.
Complicating matters, I can show that one person can hear such differences and another cannot using the exact same methodology. Is it that the difference is not audible to the latter person or that the test that made it harder for him? How do I disambiguate that as a matter of science?
In another thread, I hypothesized based on my personal experience that blind tests may provide too conservative view of audible differences. Theory I put forth was that if the mind can manufacture differences or imagine them being larger than they are, there is no reason to think that it can't do the reverse, second guessing itself in a blind test and erase a difference that may be there. And it doesn't have to do that often as to cause the results to become "statistically insignificant."
I am very interested in figuring out how to prove that real differences that are heard by the ear and the brain are indeed always detected in blind tests. Are there papers or studies I can read about this in the field of audio?
I am curious where I would read about that Steven.What he's responding to is John Atkinson claiming that its extraordinarily hard for blind tests 'to produce anything but a null result even when real audible difference exist' and it is a test that 'does not work'. Which is simply wrong.
The power of blind testing comes from elimination of bias. It does that powerfully and can be abundantly easy to see and prove using real data. Its reverse role for finding small differences is much more difficult if not impossible to prove. To wit, I can make a change to the system that is measurable and strongly so, yet not found in a blind test. The fact that we cannot use objective data to determine if our objective tests is working puts us in a tough, tough situation.
Complicating matters, I can show that one person can hear such differences and another cannot using the exact same methodology. Is it that the difference is not audible to the latter person or that the test that made it harder for him? How do I disambiguate that as a matter of science?
In another thread, I hypothesized based on my personal experience that blind tests may provide too conservative view of audible differences. Theory I put forth was that if the mind can manufacture differences or imagine them being larger than they are, there is no reason to think that it can't do the reverse, second guessing itself in a blind test and erase a difference that may be there. And it doesn't have to do that often as to cause the results to become "statistically insignificant."
I am very interested in figuring out how to prove that real differences that are heard by the ear and the brain are indeed always detected in blind tests. Are there papers or studies I can read about this in the field of audio?