Why do some Objectivists fear Psychoacousitics?

[Whatmore]

+1 Here

Your points is that blind test is not perfect .. Everyone gets that. Has gotten it for a while and on both side there will be extreme positions. For the most part objectivists are quite clear that they can'talways perform blind tests they make do with what is available. Let's use a cliche.. Almost everyone know that Blind tests are not perfect. Nothing is. yet there are tests although imperfect that yield better , more reliable results than others and on that front, sighted tests fail miserably.
As an aside it would be interesting for you to read this post by Ack:

Is your brain playing tricks on you? A case of expectation bias?



You will then make of it what you wish.

No no no you just don't get it. Here's a lesson in logic
1. blind tests are sometimes not done perfectly so we have to completely ignore all of their conclusions
2. Sighted tests are not blind
3. Therefore, sighted tests are are the source of truth.

I hope this clarifies it for everyone.

 

[jkeny]

I have read the book since a few years ago, but I also read the opinions from people who think otherwise. For example Siegfried Linkwitz :

"In this book Floyd Toole summarizes and explains conclusions from a lifelong involvement with audio. I highly recommend it to anyone interested in factual information about loudspeakers and listening rooms, about measurements, listening observations and their practical implications. It is lucidly written in easy to understand language, extensively illustrated and referenced. It deals with the reproduction of sound - which existed in a space - inside another space. My only regret is that the potential of 2-channel playback in doing so has not been fully explored and misrepresented. This is understandable because the conventional box loudspeaker with its frequency dependent directivity index has been used for almost all of the observations that are discussed. In fact, the particular interaction of a box loudspeaker with the listening room makes it more difficult for our ear/brain perceptual apparatus to hear the recording venue's space and acoustics, provided that such information has been captured in the recording process. "

Along his excellent book F. Toole often refers to articles of people who disagree with him. I find amusing that many of the followers of his "speaker rules" in WBF ignore and completely disrespect his views on room characteristics and interaction. As I have often said, IMHO Part One - Understanding the principles (that does not show in internet marketing articles) is by far the best part of the book. As always the most important is keeping an open mind, but people should first understand what is preference and the fundamentals of stereo reproduction. Yes, being audiophile is a preference.

One reason for this may well be that he uses a mono speaker test here also

This is from his blog - a test of room correction devices again done in mono. When asked why, he answered "a previous room correction study we did that found listeners were more discriminating of room correction benefits in mono versus stereo and surround."

 

[FrantzM]

John

The "Making do" I was referring to is sighted tests It is what we all do for the most part. Sighted tests. Not much time to debate that point with you. The point of picking Harman employees for their tests ... Are you implying that those tests were biased? Or if I were to use your term, "skewed"? I find this an unkind jab at people who have produce quite a respected and solid body of work...

 

[jkeny]

John

The "Making do" I was referring to is sighted tests It is what we all do for the most part. Sighted tests.

Oh, Ok, I didn't read it like that, sorry. The point I made in my reply still holds - I do think people "make do" with flawed blind A/B tests as the results suit them.

But I'm not really talking about blind Vs sighted - my distinction comes down to short Vs long-term testing. I think blind or sighted in long-term listening is not that material an issue

Not much time to debate that point with you. The point of picking Harman employees for their tests ... Are you implying that those tests were biased? Or if I were to use your term, "skewed"? I find this an unkind jab at people who have produce quite a respected and solid body of work...

Yes, I do think that picking Harmon employees as test subjects in a test that is about blind Vs sighted & using some Harmon speakers in the testing is biased/skewed whatever you want to call it - certainly not objective, to my mind. Who is going to give a non skewed result to their own company when asked to evaluate products from that company - is this going to skew the sighted results - absolutely!. Don't take one criticism of a test as a jab - take it as a point for analysis (similar to what forum debating is about - it's by trying to defend our logic that we see any weaknesses & better examine our thinking)

 

[microstrip]

John

The "Making do" I was referring to is sighted tests It is what we all do for the most part. Sighted tests. Not much time to debate that point with you. The point of picking Harman employees for their tests ... Are you implying that those tests were biased? Or if I were to use your term, "skewed"? I find this an unkind jab at people who have produce quite a respected and solid body of work...

Frantz,

This a general point we have on blind tests. They are very complex and in order to make them possible we must admit some assumptions to simplify their execution. Most of the time these assumptions can "skew" the results of the test in one direction. Scientists openly admit it - F. Toole in his book refers to it several times - and do not find opinions on it unkind or disrespectful. Again, remember that it is all based in preferences and what you are shown in the net in mostly marketing directed to members of audio forums.

 

[FrantzM]

John

What do you consider long term? How would you propose to conduct a long term (under your definition) un-sighted test?

 

[FrantzM]

Frantz,

This a general point we have on blind tests. They are very complex and in order to make them possible we must admit some assumptions to simplify their execution. Most of the time these assumptions can "skew" the results of the test in one direction. Scientists openly admit it - F. Toole in his book refers to it several times - and do not find opinions on it unkind or disrespectful. Again, remember that it is all based in preferences and what you are shown in the net in mostly marketing directed to members of audio forums.

I agree with that point if it weren't explicit in my postings. We are not dealing in perfection here ,nothing is... repetition. We are discussing the relative validity of test protocols, at least the turn this thread has taken). Some are keeping a position that since both tests are imperfect therefore both results are equally invalid. That is a red herring. Also Harman doesn't systematically take Harman employees as test subjects. And this is somewhat implied in John's post , to me a non courteous jab at the scientists behind those tests.

 

[jkeny]

John

What do you consider long term? How would you propose to conduct a long term (under your definition) un-sighted test?

I think most people know that it takes a number of days, possibly weeks to live with a new device in their audio system before they are confident of it's characteristics. I don't believe sighted/un-sighted is of great importance in this situation but what is important are the factors that are being eliminated - in short A/B testing, the different style of listening is of great importance in what is being perceived, focus, fatigue, no of repeats & limited source material are some of the issues that I can think of. All of these factors suggest that trained listeners using specific material known to reveal certain types of audio issues are the only ones capable of overcoming such hindrances.

Long term listening uses a wider selection of source material, fatigue is less of an issue. As well as that we are spreading our listening across different moods, different ear responses, different times of the day.

How do I propose to conduct such a test? It's already being done by users who listen to & report on their experience of a product over a long term. The accumulated results of those preferences is something to be taken note of. If you want something more formal I'm sure two devices can be disguised inside two identical boxes & used for a prolonged period?

 

[jkeny]

I agree with that point if it weren't explicit in my postings. We are not dealing in perfection here ,nothing is... repetition. We are discussing the relative validity of test protocols, at least the turn this thread has taken). Some are keeping a position that since both tests are imperfect therefore both results are equally invalid. That is a red herring. Also Harman doesn't systematically take Harman employees as test subjects. And this is somewhat implied in John's post , to me a non courteous jab at the scientists behind those tests.

I didn't say that - I was referring explicitly to the Harmon blind Vs sighted speaker test that is usually cited in this regard. I made no generalisations about their other tests.

 

[microstrip]

I agree with that point if it weren't explicit in my postings. We are not dealing in perfection here ,nothing is... repetition. We are discussing the relative validity of test protocols, at least the turn this thread has taken). Some are keeping a position that since both tests are imperfect therefore both results are equally invalid. That is a red herring. Also Harman doesn't systematically take Harman employees as test subjects. And this is somewhat implied in John's post , to me a non courteous jab at the scientists behind those tests.

It is not red herring, it is what naturally happens when you compare apples with oranges. Harman score cards are completely different from the typical audiophile values. And I think that John was wanting to refer to people trained by Harman own methods, but you have a point about the word employees.

BTW, it is not the tests that are invalid, it is their conclusions and the universe where they can be applied some people wisely question.

 

[TBG]

Let us say that everyone agrees that the variables involved in the relationship are broadly accepted as valid and that the independent variable explains 50% of the variance. Were people to ignore this an continue to buy what they like, what would we conclude? I grant that I doubt we will ever get to this point, however.

Are we trying to educate buyers, manufacturers, those on threads such as this, etc.? Human behavior is not as easy to assess and show relationship of note as are electrons or circuits. I studied how people vote for 40 plus years. I can easily predict how 88-90% will vote if I have them answer one question, namely 'Are you a Democrat, Republican or what?' and if they answer independent, following up with 'Do you lean toward Democrat or Republican?' Lumping leaners with those identifying with each party is high predictive. But so what? The remaining people determine who wins.

I walk into suites at shows and frequently turn around and leave given what I hear. I have often been absolutely convinced by a demonstration. I heard one of the Murata super-tweeters where those present heard good sound from a full range speaker with Muratas on top. It sounded quite good. The demonstrator then said he was removing the Muratas. He then replayed the music. Everyone almost in unison said the first music was far superior and to go back. There was some discussion of what the super-tweeters did and learned that they had no response below 15k Hz! Someone then said can we hear them alone. We were told they had been playing all the time we were discussing them. We all then listened to them more closely, and heard no music-just pops and zips. Many of us bought them on the spot, including me. I should add that they had response out to 100k Hz. I have since found that the Muratas are of no benefit with ribbon tweeters that extend out to 100k Hz.

For me hearing is believing. So is tasting wine and beer, meeting women, driving a car, watching a play, etc. Manufacturers might learn something from this, but not customers.

 

[jkeny]

It is not red herring, it is what naturally happens when you compare apples with oranges. Harman score cards are completely different from the typical audiophile values. And I think that John was wanting to refer to people trained by Harman own methods, but you have a point about the word employees.

BTW, it is not the tests that are invalid, it is their conclusions and the universe where they can be applied some people wisely question.

Let me be clear - I'm talking about using Harmon employees in this particular sighted Vs blind test taken from Sean Olive's blog headed "The Dishonesty of Sighted Listening Tests"

"A total of 40 Harman employees participated in these tests, giving preference ratings to four loudspeakers that covered a wide range of size and price. The test was conducted under both sighted and blind conditions using four different music selections."

Now please tell me if you think it wise to use Harmon employees to show how "dishonest" sighted testing is when using 3 Harmon & 1 other loudspeaker.
Do you think the sighted bias might be exaggerated by the fact that Harmon employees are the test subjects?

The 3 Harmon speakers are G, D & S Which speakers show the biggest difference between sighted & blind - G, D & S

Speaker T shows hardly any difference in preference between blind & sighted - leading to the opposite conclusion - that sighted or blind makes no difference to preference when you have no skin in the game.

I wonder why Olive heads his article ""The Dishonesty of Sighted Listening Tests" when in fact, the conditions of the test are skewed?

 

[Robh3606]

Now please tell me if you think it wise to use Harmon employees to show how "dishonest" sighted testing is when using 3 Harmon & 1 other loudspeaker. Do you think the sighted bias might be exaggerated by the fact that Harmon employees are the test subjects?

Let me be clear - I'm talking about using Harmon employees in this particular sighted Vs blind test taken from Sean Olive's blog headed "The Dishonesty of Sighted Listening Tests"

Jkeny did you even read it?? LOL


The reason for the testing all other quotes from the article in question

My mission was to introduce formalized, double-blind product testing at Harman. To my surprise, this mandate met rather strong opposition from some of the more entrenched marketing, sales and engineering staff who felt that, as trained audio professionals, they were immune from the influence of sighted biases. Unfortunately, at that time there were no published scientific studies in the audio literature to either support or refute their claims, so a listening experiment was designed to directly test this hypothesis.

Brand loyalty addressed

Brand biases and employee loyalty to Harman products were also a factor in the sighted tests, since three of the four products (G,D, and S) were Harman branded. Loudspeaker T was a large, expensive ($3.6k) competitor's speaker that had received critical acclaim in the audiophile press for its sound quality. However, not even Harman brand loyalty could overpower listeners' prejudices associated with the relatively small size, low price, and plastic materials of loudspeaker S; in the sighted test, it was less preferred to Loudspeaker T, in contrast to the blind test where it was slightly preferred over loudspeaker T.

Stereophile magazine founder, Gordon Holt, who lamented in a recent interview:

“Audio as a hobby is dying, largely by its own hand. As far as the real world is concerned, high-end audio lost its credibility during the 1980s, when it flatly refused to submit to the kind of basic honesty controls (double-blind testing, for example) that had legitimized every other serious scientific endeavor since Pascal. [This refusal] is a source of endless derisive amusement among rational people and of perpetual embarrassment for me..”

Rob

 

[microstrip]

Although, as usual in marketing propaganda, the full details are not know, what I find more interesting in the tests reported it that it was shown that in blind test conditions positional dependence is a very strong relevant parameter, with a non systematic variation with speaker model. However, as often referred, later comparative tests including competitive named speakers were carried with all speakers in the same place, disregarding any specific optimization for each type of speaker.

Many people forget that Harman tests are not absolutes for stereo reproduction, they are mostly reliable engineering tools for efficient speaker development. IMHO their preference indicator shows that under what they feel are typical conditions of use, more people will probably buy one speaker than the other. But no way that this particular speaker is a better speaker for the whole vicious WBF stereo audiophile community.

 

[esldude]

No no no you just don't get it. Here's a lesson in logic
1. blind tests are sometimes not done perfectly so we have to completely ignore all of their conclusions
2. Sighted tests are not blind
3. Therefore, sighted tests are are the source of truth.

I hope this clarifies it for everyone.

Superb post and logic. +1

 

[esldude]

I think most people know that it takes a number of days, possibly weeks to live with a new device in their audio system before they are confident of it's characteristics. I don't believe sighted/un-sighted is of great importance in this situation but what is important are the factors that are being eliminated - in short A/B testing, the different style of listening is of great importance in what is being perceived, focus, fatigue, no of repeats & limited source material are some of the issues that I can think of. All of these factors suggest that trained listeners using specific material known to reveal certain types of audio issues are the only ones capable of overcoming such hindrances.

Long term listening uses a wider selection of source material, fatigue is less of an issue. As well as that we are spreading our listening across different moods, different ear responses, different times of the day.

How do I propose to conduct such a test? It's already being done by users who listen to & report on their experience of a product over a long term. The accumulated results of those preferences is something to be taken note of. If you want something more formal I'm sure two devices can be disguised inside two identical boxes & used for a prolonged period?

This long term listening philosophy from the same guy who a few hours ago said their might be problems with the Harman test because it took 3 seconds to swap speakers? Pretty darned funny.

I agree exactly your first sentence. People get much more confident over a few days or weeks. I am pretty sure I disagree with your idea of why that is.

 

[TBG]

Let me be clear - I'm talking about using Harmon employees in this particular sighted Vs blind test taken from Sean Olive's blog headed "The Dishonesty of Sighted Listening Tests"

"A total of 40 Harman employees participated in these tests, giving preference ratings to four loudspeakers that covered a wide range of size and price. The test was conducted under both sighted and blind conditions using four different music selections."

Now please tell me if you think it wise to use Harmon employees to show how "dishonest" sighted testing is when using 3 Harmon & 1 other loudspeaker.
Do you think the sighted bias might be exaggerated by the fact that Harmon employees are the test subjects?

The 3 Harmon speakers are G, D & S Which speakers show the biggest difference between sighted & blind - G, D & S
View attachment 18517
Speaker T shows hardly any difference in preference between blind & sighted - leading to the opposite conclusion - that sighted or blind makes no difference to preference when you have no skin in the game.

I wonder why Olive heads his article ""The Dishonesty of Sighted Listening Tests" when in fact, the conditions of the test are skewed?

While this is a very small sample to draw conclusions, blinded tests seem to discourage hearing much difference between speakers. I guess I find this fascinating and hardly encouraging for blind testing, unless it more substantially correlates with what people like to hear.

 

[jkeny]

Jkeny did you even read it?? LOL

The reason for the testing all other quotes from the article in question

Brand loyalty addressed

Rob

Yes, my point is - do you think there would have been any difference if 40 people who were NOT Harmon employees were used as test subjects?
Would the difference between sighted & blind be negligible?

I don't find this a credible test of the general difference between sighted & blind & that's why I said not to cite it as evidence of this - it's more a psychology test of brand loyalty & can't be generalised to a wider context. The test should have been called "brand loyalty bias investigated" & not the very obviously skewed & biased title "The dishonesty of sighted listening" - it has nothing to do with the general case of sighted Vs blind

If I used 40 people with strong views that all DACs sounded the same & did DAC tests - do you think there would be much difference between sighted Vs blind results? Would I be wrong to extrapolate this to a wider context? Do you think if I headed the results as "Blind testing is rubbish" you would think it was a fair angle on the results or would you think I was biased in my view.

 

[jkeny]

This long term listening philosophy from the same guy who a few hours ago said their might be problems with the Harman test because it took 3 seconds to swap speakers? Pretty darned funny.

Yes, if you are doing A/B testing based on the principle of short echoic memory & your test falls outside of this time window - is it not funny? How would you defend such a test?. My stance is that I figure possible weaknesses in tests should be noted. Same with ArnyK's recent posted ABX results (null of course) - he was obviously just hitting random A or B key & not doing the test at all

I agree exactly your first sentence. People get much more confident over a few days or weeks. I am pretty sure I disagree with your idea of why that is.

No need to state the obvious, Dennis.

 

[amirm]

Let me be clear - I'm talking about using Harmon employees in this particular sighted Vs blind test taken from Sean Olive's blog headed "The Dishonesty of Sighted Listening Tests"

"A total of 40 Harman employees participated in these tests, giving preference ratings to four loudspeakers that covered a wide range of size and price. The test was conducted under both sighted and blind conditions using four different music selections."

Now please tell me if you think it wise to use Harmon employees to show how "dishonest" sighted testing is when using 3 Harmon & 1 other loudspeaker.
Do you think the sighted bias might be exaggerated by the fact that Harmon employees are the test subjects?

The 3 Harmon speakers are G, D & S Which speakers show the biggest difference between sighted & blind - G, D & S
View attachment 18517
Speaker T shows hardly any difference in preference between blind & sighted - leading to the opposite conclusion - that sighted or blind makes no difference to preference when you have no skin in the game.

I wonder why Olive heads his article ""The Dishonesty of Sighted Listening Tests" when in fact, the conditions of the test are skewed?

The whole purpose of that test was what was discovered. The story is that when Sean Olive and Toole joined Harman, they suggested using blind testing for evaluating speakers. The speaker designers wanted to have nothing to do with it. They said they absolutely knew, due to their training and skill, which speaker sounded different in sighted evaluation. So Sean tested them and they obvious failed.

Having sat through the test, I can tell you that it is an eye opener. It instantly makes a believer out of you that bias exists in evaluating speakers even though they sound so different from each other. And being associated with harman or not does not impact the results.