Do blind tests really prove small differences don't exist?

Status
Not open for further replies.

amirm

Banned
Apr 2, 2010
15,813
37
0
Seattle, WA
Admin note: this thread started at the end of jitter thread and good suggestion was made to make it its own thread. So here it is.

What he's responding to is John Atkinson claiming that its extraordinarily hard for blind tests 'to produce anything but a null result even when real audible difference exist' and it is a test that 'does not work'. Which is simply wrong.
I am curious where I would read about that Steven.

The power of blind testing comes from elimination of bias. It does that powerfully and can be abundantly easy to see and prove using real data. Its reverse role for finding small differences is much more difficult if not impossible to prove. To wit, I can make a change to the system that is measurable and strongly so, yet not found in a blind test. The fact that we cannot use objective data to determine if our objective tests is working puts us in a tough, tough situation.

Complicating matters, I can show that one person can hear such differences and another cannot using the exact same methodology. Is it that the difference is not audible to the latter person or that the test that made it harder for him? How do I disambiguate that as a matter of science?

In another thread, I hypothesized based on my personal experience that blind tests may provide too conservative view of audible differences. Theory I put forth was that if the mind can manufacture differences or imagine them being larger than they are, there is no reason to think that it can't do the reverse, second guessing itself in a blind test and erase a difference that may be there. And it doesn't have to do that often as to cause the results to become "statistically insignificant."

I am very interested in figuring out how to prove that real differences that are heard by the ear and the brain are indeed always detected in blind tests. Are there papers or studies I can read about this in the field of audio?
 

sasully

New Member
Jun 29, 2010
99
0
0
I am curious where I would read about that Steven.

You can listen to Atkinson say it, in that mp3. If its his *wrongness* you're referring to, IIRC JJ's powerpoint presentations online are a good place to start. He seems fairly convinced that DBTs using trained listeners are an excellent means to detect small differences. (Does this topic lie in the 10% of difference you have with him?)

The power of blind testing comes from elimination of bias. It does that powerfully and can be abundantly easy to see and prove using real data. Its reverse role for finding small differences is much more difficult if not impossible to prove. To wit, I can make a change to the system that is measurable and strongly so, yet not found in a blind test. The fact that we cannot use objective data to determine if our objective tests is working puts us in a tough, tough situation.

Well, define 'strongly so' . Depending on the range being measured, and the parameter being measured, a numerically large difference could still be inaudible, could it not?
Complicating matters, I can show that one person can hear such differences and another cannot using the exact same methodology. Is it that the difference is not audible to the latter person or that the test that made it harder for him? How do I disambiguate that as a matter of science?

If you can show in repeatable trials that one person can hear the difference, I'd say the difference is audible. (Scientists prefer an n of at least three.) If you can *only* find one person out of a large sample of poeple that can do that, I'd say it's audible but unlikely to be so to most people. The reason why it's inaudible to others could be multifold..another topic for investigation. It might be as simple as, one person has more acute hearing than the other. It's probably something rather mundane.




In another thread, I hypothesized based on my personal experience that blind tests may provide too conservative view of audible differences. Theory I put forth was that if the mind can manufacture differences or imagine them being larger than they are, there is no reason to think that it can't do the reverse, second guessing itself in a blind test and erase a difference that may be there. And it doesn't have to do that often as to cause the results to become "statistically insignificant."

Theories are fun,aren't they? One wonders whether they are really necessary in this case, or whether it's just special pleading -- audiophiles seem to hate the idea that their audio perception apparatus isn't a flawless detector of quality.

I am very interested in figuring out how to prove that real differences that are heard by the ear and the brain are indeed always detected in blind tests. Are there papers or studies I can read about this in the field of audio?


Well, how would you determine that differences are 'heard by the ear and the brain' without a listening test? fMRI? And is physiological activity detected in the ear and brain the same as 'heard'? Even in the visual system, our eye 'sees' lots of things that the brain edits out.

And by 'always' detected in blind tests, are you suggesting that all blind tests are equally well-constrained? Or are you referring to a particular protocol, e.g., one proposed as AES standard?
 
Last edited:

amirm

Banned
Apr 2, 2010
15,813
37
0
Seattle, WA
You can listen to Atkinson say it, in that mp3. If its his *wrongness* you're referring to, IIRC JJ's powerpoint presentations online are a good place to start.
I meant the latter. I have watched JJ's presentations but I am not sure to completion or inclusion of this point. Do you remember his explanation and logic? And can you talk it through in your own mind?

He seems fairly convinced that DBTs using trained listeners are an excellent means to detect small differences. (Does this topic lie in the 10% of difference you have with him?)
No, I have not had this discussion with him. Wouldn't be surprised at all that he would have this position but I don't know the reasoning he might have.

Well, define 'strongly so' .
As in completely changing the psychocaoustics model of a codec and hence all the bits changing yet the outcome not being conclusive at all that anything changed. Take this simpler example. We tested 2:1 compression of music from 1.4 mbit/sec to 750 kbps. In doing so, all the samples are changed as the other example yet detectability was exceptionally low. I could tell the difference and I think a few others but tens of others could not.

Yes, the codec does an excellent job of hiding its artifacts but quandary remains: we know we changed the samples drastically. We know some people heard it. But many, many people could not. How does one disambiguate whether the few of us had better listening ability vs blind testing disadvantaging people who are not used to running it?

Depending on the range being measured, and the parameter being measured, a numerically large difference could still be inaudible, could it not?
Of course. I am looking for a protocol to prove that and find that it gives me a headache :). I am often fascinated how astrophysicists (?) think of clever schemes to figure out the existence of celestial bodies light years away based on gravity, X-ray and such and wonder out lout if there is way to create an experiment that quantifies this issue.

When we use an instrument, we know its accuracy level. It seems odd that we use blind listening tests yet we don't know how much they could misfire. Are they wrong 10%, 20%, 30%, 50%? I have had third-parties run blind tests that we knew to be 100% wrong and indeed, they arrived perfectly wrong conclusions! In those cases, we knew the data set they used was not revealing of what was being measured. But in many of these other cases we are in the dark.

If you can show in repeatable trials that one person can hear the difference, I'd say the difference is audible. (Scientists prefer an n of at least three.) If you can *only* find one person out of a large sample of poeple that can do that, I'd say it's audible but unlikely to be so to most people.
I don't know about the last part of this sentence. I would say that is an assumption that I like us to challenge and understand. I want to know why the other people did not hear it. We can't say we believe in objective testing yet not have an objective and positive proof point for that.

Well, how would you determine that differences are 'heard by the ear and the brain' without a listening test? fMRI? And is physiological activity detected in the ear and brain the same as 'heard'? Even in the visual system, our eye 'sees' lots of things that the brain edits out.
I have asked this question before. We have a number of people in the medical field here so perhaps some suggestions can be made. My sense however is that medical measurements are nowhere accurate enough to detect these differences but I am just guessing.

And by 'always' detected in blind tests, are you suggesting that all blind tests are equally well-constrained? Or are you referring to a particular protocol, e.g., one proposed as AES standard?
I am referring to all of them. Here is a scenario:

You are comparing two samples. You hear a difference. Then you think, maybe I am being tricked. After all, this is a hidden reference in there and I sure don't want to look like a fool not hearing it :). Or if you don't know that, you wonder based on previous samples that sounded the same, this one is probably the same and you are imagining it. So you try to convince yourself that there is no difference and lo and behold, there isn't any and you vote that way.

Now what if it turns out that there was a difference after all that others detected? Did we just get a more pessimistic of this current tester because we subjected him to a test protocol?
 

JackD201

WBF Founding Member
Apr 20, 2010
12,308
1,425
1,820
Manila, Philippines
From my perspective the answer is no, blind tests do not prove small differences do not exist. Why? Because the output of blind tests are statistical. Being such, there will always be the probability of different results for the same test when performed on different sets of subjects. One group may have outliers while others might not. Given the identical test in other words, 1+1 may or may not be equal to 2.

It is still an invaluable tool however when biases need to be erased. Like Amir says, bias goes both ways. There are positive and negative biases. Sean tells us it takes time to get reliable results from a testing panel on small differences. It takes training and experience. The subjects have to be so used to taking blind tests that there is no anxiety about having to prove anything about themselves one way or the other.
 

terryj

New Member
Jul 4, 2010
512
0
0
bathurst NSW
From my perspective the answer is no, blind tests do not prove small differences do not exist. Why? Because the output of blind tests are statistical. Being such, there will always be the probability of different results for the same test when performed on different sets of subjects. One group may have outliers while others might not. Given the identical test in other words, 1+1 may or may not be equal to 2.

Ok, what about the more *usual* type of test (hmm, IS it more usual?).

Specific guy A says 'I hear yada yada'.which according to the other side is 'debatable' at least.

So Specific guy A does the test, in his own home, on his own gear and now fails to hear what he said he heard.

Ok maybe that is not so usual at all, but as a thought experiment how far does it go to answer your considerations??

Amir, I appreciate what I think is the thinking behind this, will re-read it again but just whilst answering now-but I can't help thinking that it is starting from the wrong point/for the wrong reason?

At least from my perspective.

It's kinda a chicken and egg thing maybe, 'DBTs have trouble revealing the small differences' only has validity IF you believe those small differences are real.

'DBTs have trouble revealing small differences' has no validity if you feel those small differences do not exist.

Haha, maybe the old truism is true, the truth lies in the middle somewhere?...(yes, I get that is what you are asking)

You are asking for papers etc on this?? Would there be any?

The counter to your example where we know the signal was markedly different yet not heard, are the ones where we know there was NO change in the signal, yet a difference was heard. (bias, already acknowledged).

Personally I suppose these sort of musings give me direction on how to view it all. IF actual real differences are NOT heard in a dbt, then (for me) it is ok to completely ignore them. That is not an answer to the question, but I feel it is a useful way to use the results of dbt's.

The other way to look at Jack's point (1+1 does not always equal two) is 'who cares' if someone, somewhere eventually can hear the difference between two cables. Yet every audiophile cable believer out there WILL think 'a-hah, I can hear it too'. I do not hold my hearing in so high a regard to suppose I have that ability.

I know people can run the 100m in <10 secs, the perception of my self worth does not require me to think I can do it too.


That IS all off topic to a degree, yet in some way is an answer to the question of the worth of dbt's.
 

amirm

Banned
Apr 2, 2010
15,813
37
0
Seattle, WA
Here is some more food for thought and what I have experienced in real life, taking these tests.

Let's assume there is a spectrum from 1 to 100. At 1, everyone hears a solid difference with 100% confidence. And that 100% the opposite is true, no one hears anything no matter what.

Let's further assume that the transition point is at 70 when thing start to tilt from one direction to the other.

What happens to your evaluation of what you are hearing as you climb toward 70? We know from electronic circuits called comparators that they become unstable in that area and their function ceases to work. My brain does the same thing. It thinks it hears something but can't swear by it. So it vacillates especially when I am being tested and have to take sides.

My theory is that blind tests are unreliable when used in that transition point. Our ear and brain combination are simply not designed to give binary answers in that area that the test requires.

Working backwards, we know that differences do exist in transition area because that is the definition of transition area. It is a gray region between no difference and a difference.

For mass market products, we don't care about transition area. We can opt to assume that any uncertainty is not worth the cost in equipment cost to chase out. This is important as these discussions do not erupt with average consumers but audiophiles. For them, the extra cost is justified if it rules out audibility issues even some of the time (think of a few millisecond transient of a piano or guitar pick).

To wit, I rip into lossless format because no matter how high I set the bit rate of a lossy codec, once in a while I hear an annoying artifacts for a fraction of a second and even if the rest of the song is perfect, that bothers me.

The above is why I like to know how we can quantify where in that spectrum the test is. I look at blind tests and I see someone just getting some random "audiophile" content and assume it is revealing of the issue at hand (jitter, high resolution formats, etc.). We never do that in other proper areas of audio research. For compression tests, we use tracks that are hugely revealing of artifacts. This pushes us past the 70% point above.

Using trained listeners also helps to push us out of this hazy area as mentioned because they are able to hear things more readily and less inclined to second guess themselves.

All of this is compounded by induction logic people want to use to take the results of one blind test, and apply it to all people, all content and all equipment. We know that no test is applicable that way when it produces negative results. It is tempting however to say, "well, 500 people couldn't hear the difference statistically therefore no one can." I say before you can say this, you at least need to demonstrate if you were or were not in the 70% area. Because if you are in 70%, your testers were "oscillating" in how they were voting which is all that is needed to invalidate their results beyond chance.
 

terryj

New Member
Jul 4, 2010
512
0
0
bathurst NSW
Here is some more food for thought and what I have experienced in real life, taking these tests.

Let's assume there is a spectrum from 1 to 100. At 1, everyone hears a solid difference with 100% confidence. And that 100% the opposite is true, no one hears anything no matter what.

Let's further assume that the transition point is at 70 when thing start to tilt from one direction to the other.

What happens to your evaluation of what you are hearing as you climb toward 70? We know from electronic circuits called comparators that they become unstable in that area and their function ceases to work. My brain does the same thing. It thinks it hears something but can't swear by it. So it vacillates especially when I am being tested and have to take sides.

My theory is that blind tests are unreliable when used in that transition point. Our ear and brain combination are simply not designed to give binary answers in that area that the test requires.


Got the concept, but WHY does that happen? The starting point you are making is that because it is a DBT, when we reach the 70% mark we oscillate.

IF the stimulus is at the 70% mark, why does it not apply in a non dbt situation? Why in a sighted audiophile condition does oscillation not occur?

In other words, why is the sighted conclusion more reliable?

The above is why I like to know how we can quantify where in that spectrum the test is.

For sure, you gotta test the correct thing!

All of this is compounded by induction logic people want to use to take the results of one blind test, and apply it to all people, all content and all equipment. We know that no test is applicable that way when it produces negative results. It is tempting however to say, "well, 500 people couldn't hear the difference statistically therefore no one can." I say before you can say this, you at least need to demonstrate if you were or were not in the 70% area. Because if you are in 70%, your testers were "oscillating" in how they were voting which is all that is needed to invalidate their results beyond chance.

I agree that these conclusions are pushed past their comfort zone, but am wary about the 70% area you are emphasising again. Again, if we cannot overcome this 'problem' with a dbt, I am curious what argument there might be that normal, non-rigorous audiophile listening gives more reliable results.

The point may be valid, but be wary about throwing out the baby with the bathwater by going to usual auditioning procedures???
 

JackD201

WBF Founding Member
Apr 20, 2010
12,308
1,425
1,820
Manila, Philippines
Ok, what about the more *usual* type of test (hmm, IS it more usual?).

Specific guy A says 'I hear yada yada'.which according to the other side is 'debatable' at least.

So Specific guy A does the test, in his own home, on his own gear and now fails to hear what he said he heard.

Ok maybe that is not so usual at all, but as a thought experiment how far does it go to answer your considerations??

Then I learn more about my personal limits don't I? I can't see how that can be a bad thing. Still, saying I tested myself and failed doesn't mean everybody will fail too. I'd fail Amir's artifact test for the simple reason I can't put a name to the artifacts. If I don't know what I'm listening for and don't know what to name it, what answer could I give that could be correlated by Amir? If I did hear artifacts then I'd have to resort to using whatever language I have available using words like edge, blur or whatever nebulous descriptor the objective types absolutely can't stand because of the lack of specificity. Is that the subject's fault? I say hell no. That's the tester's fault.

The great irony of DBTs is that those that herald it, sometimes to the point of being the end all and be all, always say things like "I do not hold hearing in so high regard". Well, DBTs are listening tests. Hearing can't be that bad then. DBTs have and will always be used for verification of audibility. That's it. The output will always state the confidence intervals. At a confidence interval of 95, one could have counter occurrences upwards of 35%. Statistics is a science but one must remember the math is quite a bit different in the way it is appreciated by the layman. There are many cases of wrongful convictions based on expert testimony misunderstood by the juries. 1 + 1 is not equal to two a lot of the time just as a confidence level of 95 does not guarantee 95% outcome.

Now I have absolutely no problem accepting the results of DBTs for as long as the conclusion presents the statistical parameters. I will not however accept the outcome as absolute for reasons I hope I have made clearer.
 

Ron Party

WBF Founding Member
Apr 30, 2010
2,457
13
0
Oakland, CA
Jack, nobody who subscribes to the validity of blind tests says that to which you ascribe. Far from it.
 

amirm

Banned
Apr 2, 2010
15,813
37
0
Seattle, WA
Got the concept, but WHY does that happen? The starting point you are making is that because it is a DBT, when we reach the 70% mark we oscillate.
Oh, let me clarify. The 70% had nothing to do with DBT. I picked that as the demarcation point where the probability of difference tilts the other way.

IF the stimulus is at the 70% mark, why does it not apply in a non dbt situation? Why in a sighted audiophile condition does oscillation not occur?
Let's say it is applied to both. If in DBT I detect the difference half the time and not the other, the result is a coin toss. But if that is what I am supposed to do at the boundary, then it is not chance. The half was indicative of real difference heard.

In other words, why is the sighted conclusion more reliable?
I tried to explain some of the reasons. One is the notion of second guessing yourself since it is a test and we are trained by schooling and such to try to "pass a test." As soon as there is a test, then there is a desire to be right and that can distort things. At least it has with me where I second guessed myself, just to find out that I shouldn't have as there was a difference and others voted that way.

The system works when the difference is large or absolutely does not exist and can be shown to mathematically to be so. Here, we have devices under test that mathematically are different. Yet the difference audibly is small enough to be in boundary so neither one of these conditions apply.

The point may be valid, but be wary about throwing out the baby with the bathwater by going to usual auditioning procedures???
Let me say that all of this is a theory on my part, hoping some protocol can be discovered to prove or disprove it.
 

amirm

Banned
Apr 2, 2010
15,813
37
0
Seattle, WA
Amir are you not casting doubt on the whole concept of JND?
I believe in it but the question is, was that limit determined in blind testing? If so, then it is a circular argument in my opinion.

In other words, what if blind testing underestimates JND by 50%? How do we know it doesn't based on the factors I mentioned?
 

terryj

New Member
Jul 4, 2010
512
0
0
bathurst NSW
Then I learn more about my personal limits don't I?

Was a general observation, not particularly directed at you, but at the idea you put forward that DBTs don't find the small differences because they are statistical, ie one positive can outweigh a negative.

I guess some of this CAN come back to the honesty or otherwise of the people doing the test. Someone in a crowd might tell amps apart say. However IF they were lumped into the entire total mix, then yeah statistically it comes out at 50%.

We rely on the honesty of the conductors to point out that *one* person DID hear a difference statisically yada yada. Maybe take him aside and do further testing.

So that is the general, massed group thing I think you were referring to. The example I gave was the counter to the 'general', hence I used Mr Specificexample.

I would tend to rely on the results if there were many examples of Mr Specific, each completely sure (for example) he can tell cables apart. Each time a test happens where he was unable to pick them tends toward the same answer, science is perhaps correct. (don't want to get into THAT argument here, so take it as it is meant)

I can't see how that can be a bad thing. Still, saying I tested myself and failed doesn't mean everybody will fail too. I'd fail Amir's artifact test for the simple reason I can't put a name to the artifacts.

Would that not come under the heading of a 'bad test'? At the very least the person should have enough education to do the test?

Again, with the Mr Specific example, he would not need to be able to describe what he hears, he just needs to be able to hear it and say when 'it' appears. That is sufficient.

Is that the subject's fault? I say hell no. That's the tester's fault.
Should have read it before I answered, Yep, agree with you, see above.

The great irony of DBTs is that those that herald it, sometimes to the point of being the end all and be all, always say things like "I do not hold hearing in so high regard".

Do they? I feel that you are already setting it up to knock it down.


Now I have absolutely no problem accepting the results of DBTs for as long as the conclusion presents the statistical parameters. I will not however accept the outcome as absolute for reasons I hope I have made clearer.

Ahh, I think you felt I was challenging you. Not really, I WAS just asking you if the Mr Specificexample countered your objection of the statistical nature.

Oh, let me clarify. The 70% had nothing to do with DBT. I picked that as the demarcation point where the probability of difference tilts the other way.

Don't worry, I get the point that what is important is the oscillation bit, not the number.

Let's say it is applied to both. If in DBT I detect the difference half the time and not the other, the result is a coin toss. But if that is what I am supposed to do at the boundary, then it is not chance. The half was indicative of real difference heard.

I think we are 'running around in circles' here. One part is definitional maybe, in this sentence above for example you say he was supposed to detect a difference, yet the way you worded it showed that it was only detected 50% of the time.

If only detected 50% of the time, is it a yes or a no? (maybe that IS your essential point?) Heck, dunno, tho it does seem rather strangely setup.


I tried to explain some of the reasons. One is the notion of second guessing yourself since it is a test and we are trained by schooling and such to try to "pass a test." As soon as there is a test, then there is a desire to be right and that can distort things. At least it has with me where I second guessed myself, just to find out that I shouldn't have as there was a difference and others voted that way.

Is the problem then an historical one? On hindsight, would it have been better to call it a double blind investigation? I don't quite buy the idea that the mere fact of labeling it a test suddenly introduces all this angst, worry and fear, and intense burdens of needing to pass.

Even so, and this goes back to some of the points with jack above, that is definitely the fault of the person conducting the test?? It should be done as many times beforehand top get familiarisation etc, becomes calm and in control of proceedings (the change only occurs when the person says so etc etc). In most cases (?) that usually happens does it not?

Having said that, I do think there IS a sudden level of pressure created, usually the first time a switch is made when the person has become blinded to the identity. That is when it suddenly hits them that these HUGE differences seem to melt away.

So I don't quite buy the stress angle.

The system works when the difference is large or absolutely does not exist and can be shown to mathematically to be so. Here, we have devices under test that mathematically are different. Yet the difference audibly is small enough to be in boundary so neither one of these conditions apply.

That is begging/pre-empting the question...
 

LL21

Well-Known Member
Dec 26, 2010
14,411
2,509
1,448
Personally, i absolutely find some people (all of us to some degree, perhaps) are open to the power of suggestion...and end up hearing what we want to hear. I will say this however...there are 2 ways to listen to a system when auditioning a component. 1) get out the checklist, do i hear the "ting"? check. do i hear the echo? check. 2) r-e-l-a-x...do i enjoy the music and feel it is more natural? In a DBT, i find most people seek to get it right by focusing on 1. It is a crutch because the checklist is more defined...but that is NOT always where we are hearing differences. In the end, i know the DCS Scarlatti has superior technical check-the-list technical performance than my Zanden. But i never stop enjoying listening to music with the Zanden and that is the difference for me which is not so easy to get out of a test...because generally you are switching and focused on "i can spot the component". But i think careful listeners who might not do well on such a test...might be relieved to find that take either component home for 3 weeks...and their preference might become more definite and consistent.
 

Phelonious Ponk

New Member
Jun 30, 2010
8,677
23
0
Now I have absolutely no problem accepting the results of DBTs for as long as the conclusion presents the statistical parameters. I will not however accept the outcome as absolute for reasons I hope I have made clearer.

There are no absolutes, of course. That is not an excuse for believing whatever we hear with our eyes, expectations and desires fully engaged and dismissing all evidence that challenges it. And personally, I see a lot more of that in the audiophile world than belief in the absolute conclusions of DBT.

Tim
 

JackD201

WBF Founding Member
Apr 20, 2010
12,308
1,425
1,820
Manila, Philippines
Don't get me wrong. Blind tests are extremely important especially when it comes to bringing value into products. I mean what's the point in putting a particular component in a product if the market their intended for isn't sensitive to what it brings to the table. So no I am not setting things up to tear it down at all. I firmly believe that everybody should experience a DBT at least once in their lifetime if only to get the hubris knocked out of us. Let it be a fair test though. DBTs are extremely easy to use as a way to knock people down in a mean spirited way just to prove some point. That isn't just a bad test, that's a malicious test. What we will find from the most reputable manufacturers is that DBTs are an integral part of the testing process but by no stretch the only tests performed. Along with the blind tests are empirical measurements as well as sighted tests.

There was one former member here that went so far as to categorically say that his own DBTs which by his own admission did not live up to the most rudimentary standards were the end all be all and even said that correlation of data wasn't even needed. On the world wide web, he isn't alone.

Then there are guys like FrantzM who DBT'd himself and ended up saving himself lots of dough while maintaining his sonic happiness. Frantz is to me a good example. Frantz never preached. He just shared his experience that others might benefit the same way if they so chose. The other guy mentioned, was to me a bad one. He played some sort of savior wannabe but his science sure was shoddy.

In my first post I did qualify that I was speaking only from my perspective. That would be the perspective of someone who loves statistics. As someone who loves statistics, I know and respect its limitations. I also have a pretty good handle on their applications and misapplications.

As far as the topic of this thread goes, I stand my ground. A statistical test based on perceptions cannot empirically disprove a change or difference in variables be it sighted or not. All these tests will tell us is if these changes or differences can be detected and identified given certain conditions. These are two very different things. I think we can all agree on that.
 

Ron Party

WBF Founding Member
Apr 30, 2010
2,457
13
0
Oakland, CA
A few obvious points. First, it is not circular, unless you subscribe to the notion that the scientific method itself is invalid, i.e., the perceptual world cannot be studied. If it can be studied, the scientific method is THE way.

Second, and I know you know this, but it must be emphasized that DBTs don't *prove* a difference does or does not exist. Don't lose yourself in your on-line battles with A.J. & Arny and lose sight of the basics.

Third, if you know of a better way to study the subject at hand which is even remotely as reliable, the entire scientific community is, well, all ears.
 

amirm

Banned
Apr 2, 2010
15,813
37
0
Seattle, WA
Third, if you know of a better way to study the subject at hand which is even remotely as reliable, the entire scientific community is, well, all ears.
The "better" way is actually what is occurring in real life. It is a mix of measurements, subjective evaluation with expert listeners and occasional blind tests to rule out placebo effect. This is how and everyone else that I know develops audio technology.

It is only in forums that people think everything has to be done with double blind ABX tests or nothing.

So in that sense, whatever danger there is in blind tests underreporting performance improvements, it is countered by the expert subjective testing.
 

amirm

Banned
Apr 2, 2010
15,813
37
0
Seattle, WA
First, it is not circular, unless you subscribe to the notion that the scientific method itself is invalid, i.e., the perceptual world cannot be studied. If it can be studied, the scientific method is THE way.
I am confused how this relates to the question you asked me. You asked does JND not apply. I said that if you use blind testing to establish your JND, then I can't use that as a assumption of what the JND is in reality outside of blind tests. Put another way, if I accept that JND has to be established with blind testing, then the next question of whether small differences are audible in blind tests is moot!

We need to establish a new protocol to get there. Using existing protocol doesn't work to prove or disprove the hypothesis at hand.
 
Status
Not open for further replies.

About us

  • What’s Best Forum is THE forum for high end audio, product reviews, advice and sharing experiences on the best of everything else. This is THE place where audiophiles and audio companies discuss vintage, contemporary and new audio products, music servers, music streamers, computer audio, digital-to-analog converters, turntables, phono stages, cartridges, reel-to-reel tape machines, speakers, headphones and tube and solid-state amplification. Founded in 2010 What’s Best Forum invites intelligent and courteous people of all interests and backgrounds to describe and discuss the best of everything. From beginners to life-long hobbyists to industry professionals, we enjoy learning about new things and meeting new people, and participating in spirited debates.

Quick Navigation

User Menu

Steve Williams
Site Founder | Site Owner | Administrator
Ron Resnick
Site Co-Owner | Administrator
Julian (The Fixer)
Website Build | Marketing Managersing