Objectivists, Harman Testing, Reviewers, and Reality

[amirm]

I agree, I just think it was a well written opinion on the subject. Personally, I'm all for both measurements and subjective listening and I think anything that can be done to further the correlation of measurements to how we perceive sound is a step in the right direction. Amir has made some irrefutable points imo, but it's also true that as complex as the subject is, it's still nowhere near complex enough to completely define how sound is perceived. It would be interesting to see if Harman has managed to measure more obscure things like detail or resolution of a speaker, speed or PRAT, cohesiveness, etc... we know they won't share information if they think withholding it would be a competitive advantage so it's hard to know exactly how much progress has been made in this area.

No concern about commercial conflict of interest as Dr. Toole addresses these other performance vectors in his 1986 JAES paper when he worked for NRC. Unfortunately papers that old are scanned archives which means I literally have to type all of them in to quote, or take screen snapshots. Here is a short summary of it as a snapshot:

The text leading up to this says while the other characteristics are complex and important, they routinely follow along with proper behavior in frequency domain. I still remember the first time I reduced a peak in bass frequencies and all of a sudden, I could hear so much more detail in higher frequencies. The resonance in time was smearing the other details, especially since bass is so powerful relative to mid and high frequencies. This is totally non-intuitive but anyone who has experimented with EQ knows it 100%. So the first job is to produce a loudspeaker that has well behaved frequency response. Once there, yes, as I indicated in my earlier post, there are other characteristics that you want to detect by auditioning. What you don't want to do is jump into those without dealing with the major factor involved.

As I have noted, loudspeakers that have proper characteristics in frequency domain, also are room friendly. The reflections are similar to the direct sound so having them there or not makes far less difference than loudspeakers that have large differences as many do. So that is another reason to look at frequency response measurements.

I also think there's room in the world for both subjective reviews as well as measurements. I'd agree most people don't care and aren't trained to understand measurements, but some are and some people obviously find them valuable. If you are a reviewer trying to reach the broadest possible audience a review that covered both areas thoughtfully and thoroughly would probably be the best way to go,

I think beyond the reviewer writing something for us, they need to educate themselves. They need to be familiar with all of this research. Perform their own measurements and try to correlate them to what they hear. They need to become trained listeners. Only then will they be in a superior position to us as readers to convey information that can be taken as authoritative and informative. To the extent they are a peer reviewer to the rest of us, they have substantially lowered the bar for what a reviewer is supposed to be, and do.

 

[Atmasphere]

Here lies the ultimate problem with your approach. Full sets of measurements are facts. It's the subjective, poetic nonsense that is fiction.

Tim

Looks like you missed my post- full measurements are not in fact, facts. Not if the equipment has design errors and not if the operator does not have enough knowledge to recognize when there is a problem.

I can give you several examples that come from direct experience. One of them I already mentioned- that the Audio Precision tester used to test one of our preamps at Stereophile did not conform to AES 47 (our preamp OTOH did; the result was that the output of the preamp 'measured' at lot noisier than it really was when the bug was actually with the test equipment and a lack of knowledge on the part of JA). Another I have seen several times is that when testing our amps, the speaker terminals must not be allowed to be shorted to ground. The result will be lower power and higher distortion (due to imbalancing of the drive to the power tubes). I've seen that four times so far. Yet these specs get published as if they are real.

So the 'objective' aspects can be fictional too.

 

[mep]

Amir-you keep going back to a $60 setup to measure speakers and seem to have forgotten that yesterday you said all reviewers need to measure everything they review which entails a hell of a lot more money in test gear than $60. You even mentioned that your test gear is worth $60k-$80k.

Now back to that $60 setup... JA uses the DRA Labs MLSSA system which costs around $4,300. JA uses two microphones; a Calibrated DOA 4006 which is around $2,000 and an Earth Works QTC-40 which sells for $1,249.

 

[amirm]

Amir-you keep going back to a $60 setup to measure speakers and seem to have forgotten that yesterday you said all reviewers need to measure everything they review which entails a hell of a lot more money in test gear than $60. You even mentioned that your test gear is worth $60k-$80k.

No, not for loudspeaker measurements. The $60K set up was for digital audio characterization which is a different topic. You only need $60 and some knowledge to be way, way ahead of the game.

Now back to that $60 setup... JA uses the DRA Labs MLSSA system which costs around $4,300. JA uses two microphones; a Calibrated DOA 4006 which is around $2,000 and an Earth Works QTC-40 which sells for $1,249.

MLSSA used to be cat's meow before advent of free programs like REW. People who have them still use them but is not necessary or even good. You can get a calibrated version of the $60 microphone for $100 from Cross-spectrum. It won't be NIST traceable, or have the pedigree of B&K, but it is more than good enough for making these types of measurements. All you have to do is document what you are using.

But again, I assure you that if you can get loaner $50,000 loudspeakers, you can get loaner $1,200 microphones if you give them a mention in your review footnote. It is the motivation and knowledge which you must bring yourself. The rest is not a barrier. I know if I decided to write those reviews, I could get them in a heartbeat.

 

[mep]

Hello Mep

That's not really true. You don't need professional measurements or equipment. What you need to do any meaningful comparisons are measurements done under identical test conditions. You need repeatability in the same set-up and from set-up to set-up or it all goes out the window. 12 Reviewers 12 different measurement set's which may look different but actually be correct because of placement of the microphone and distance, gate times chosen which is a big one, and various other reasons. None of these guys would have access to an anechoic chamber so it would be a nearfield splice to a gated measurement like JA does in Stereophile. This is a compromise and will not match anechoic measurements of the same speakers done by JBL/Harman s an example.

Rob

Rob-you are being like Amir and forgetting that he wasn't just talking about testing speakers, he was talking about testing electronics too. And regardless of what Amir says, JA is not using $60 worth of test gear to measure speakers. It's more like $7500 or so.

 

[NorthStar]

If you want a full set of professional measurements, then you you need people who are trained professional technicians or engineers who have professional test equipment. And now you also want the reviewers to have access to the measurements prior to writing their reviews and you don't think that will change how reviews are written? That comes with its own perils because now you could see reviews written around the measurements and not written about how the gear sounds.

I think we are quickly approaching the point on this forum where some people think they could place blind faith in measurements (if they really had them, and yeah, I've been told it's my job to go get them) and they don't want or need to read reviews.

Hi Mark,

Michael Fremer, among other audio reviewers, write his reviews, then the audio gear is send to the lab test to be evaluated/measured by the experts with their best measuring tools and mics in measuring all important aspects of the gear that has been already reviewed.
Sometimes it coincides with the reviewer's ears, and sometimes not @ all.

For me, that's the type of audio reviews that I'm interested in; I can see clear, I can see almost all. ...Objective reviewers without a single drop of bias; truly professionals and integre in their line of audio passion/work/hobby...

 

[amirm]

Looks like you missed my post- full measurements are not in fact, facts. Not if the equipment has design errors and not if the operator does not have enough knowledge to recognize when there is a problem.

I can give you several examples that come from direct experience. One of them I already mentioned- that the Audio Precision tester used to test one of our preamps at Stereophile did not conform to AES 47 (our preamp OTOH did; the result was that the output of the preamp 'measured' at lot noisier than it really was when the bug was actually with the test equipment and a lack of knowledge on the part of JA). Another I have seen several times is that when testing our amps, the speaker terminals must not be allowed to be shorted to ground. The result will be lower power and higher distortion (due to imbalancing of the drive to the power tubes). I've seen that four times so far. Yet these specs get published as if they are real.

So the 'objective' aspects can be fictional too.

This is true too. And very difficult to remedy at times due to reviewer reluctance to measure things differently than other equipment or understanding the need why.

 

[Atmasphere]

^^ The funny thing is you can explain it to them and they might get it. But the publishing industry being what it is, often this happens just before publication. That is how it went down with Stereophile. If I had not called, I would not have found out about the measurement gaff at all- Stereophile did not even FAX me a galley proof. It was not as if they even changed a word in the article...

This is one of the reasons I maintain that bad reviews are unethical.

 

[Atmasphere]

Exactly. Audio Research has often had the same problem (its not an OTL thing, its that the output has equal impedance to ground from either speaker terminal).

Even so that does not mean that someone testing will get it right. Its very easy to short the output of the amp through the test equipment, which usually has the ground pin of the power cord plugged into the wall. Its that connection that is often the culprit. It can make the difference of the amp having 5-10% THD at full power or only 0.5-1%!!

 

[DaveC]

Since you asked, speed and so called PRAT are all about the entire box of speakers individual speakers (woof, mid tweet etc) each launching their specific band of the signal at the same time it occurs at the input terminals of the speaker, and that is measured all the time. There is no other interpretation of speed than the measurement I talked about. However, these elements can get screwed up way before they get to the speaker terminals, but it is only fair to measure at the input to the speaker if we are talking about the speaker.

I'd put time-aligning drivers under cohesiveness although it also does have something to do with the perception of speed/PRAT.

There's also the Mms, Bl and inductance of the driver(s) to consider for the perception of speed as well as the effects the chosen cabinet design might have on the driver's performance. This is evident by the difference in perceived speed/PRAT of crossoverless single driver speakers, which can vary quite a bit in this area despite not having to deal with the effects of crossovers and multiple drivers.

 

[esldude]

I repeatedly see how measurements don't tell all, and don't match up with what we hear subjectively. And on and on with excuse after excuse why repeatable measurements aren't telling enough. I think much of that complaint is caused by sighted listening. If you did more blind listening, I think you will find pretty good agreement with even basic measurements. We know (yes know, you, whomever you are, aren't immune) sighted knowledge clouds simple perception. By blind listening I mean simply listening without knowing what you are hearing. Not blind ABX testing or such. Just listen without knowing what the DUT is.

So how could that work in reviewing speakers? You have to blind the speakers from the reviewer. One way would be for a magazine to maintain a good listening room. Put speakers under review behind a transparent curtain, and let one or more reviewers spend all the time they want listening. Followed by writing up the review. They wouldn't know what was behind the curtain, and would have only their perceptions to go by. Or they could place such a curtain over one end of their personal listening room. Have someone come and do a setup behind the curtain when they aren't at home. If this were done, I think there would be much less disagreement of sound quality vs measurements of that sound.

Short of that, what is more reliable, repeatable and useful. Measurements by dumb instruments or flowery subjective reviews by human reviewers who are effected by the product under review even before they hear it?

 

[microstrip]

(...) Interestingly, it appears to me that a somewhat middle ground has appeared, that elucidated by Alan Sircom, but of course I may be misreading his intentions as wel...

Yes, those who understand that both measurements and subjective reviews have intrinsic limitations, that both need a lot of knowledge, understanding, familiarization and experience from the reader. That most of the consumers (either those who support the measurements or the subjective reviews ) are not interested in spending their time to achieve proficiency in reading any type of review, but enjoy their preferred type.

Real life is never simple. The high-end stereo reproduction creates a form of culture, centered around a characteristic way of listening to sound reproduction that deeply involves the listener. It evolves around brilliant designers, manufacturers and dealers that create and sell products that their owners appreciate and enjoy. A few fail this objective, we must accept it. Subjective reviews and reviewers are part of it. Measurements can be part of it if they can be correlated with the nuances of the high-end. Until they are not, we must avoid throwing out the baby with the bath water.

As Nelson Pass wrote:

If you are concerned that your power amplifier (or anything else
for that matter) is as objectively and technically accurate as possible,
that is a perfectly legitimate criterion. You will certainly find many
products in the marketplace that excel at conventional objective
performance, and most of them are much cheaper.
Our real customers care most about the experience they get when
they sit down to listen to their music. We create amplifiers that we
like to listen to, on the assumption that we share similar taste.
We want our products to invite you to listen. We want you to enjoy
the experience so much that you go through your entire record
collection - again and again. This, by the way, is a very strong
indicator.

 

[microstrip]

(...)

Short of that, what is more reliable, repeatable and useful. Measurements by dumb instruments or flowery subjective reviews by human reviewers who are effected by the product under review even before they hear it?

Reliable - none of them
Repeatable - measurements
Useful - the subjective review

Just MHO, YMMV

 

[rbbert]

...As Nelson Pass wrote:

If you are concerned that your power amplifier (or anything else
for that matter) is as objectively and technically accurate as possible,
that is a perfectly legitimate criterion. You will certainly find many
products in the marketplace that excel at conventional objective
performance, and most of them are much cheaper.
Our real customers care most about the experience they get when
they sit down to listen to their music. We create amplifiers that we
like to listen to, on the assumption that we share similar taste.
We want our products to invite you to listen. We want you to enjoy
the experience so much that you go through your entire record
collection - again and again. This, by the way, is a very strong
indicator.

Coincidentally I'm getting a new INT60 for in home trial starting next Tuesday

 

[Atmasphere]

Reliable - none of them
Repeatable - measurements
Useful - the subjective review

Just MHO, YMMV

The way I see it:

Reliable - none of them
Repeatable (sometimes, if done correctly) - measurements and subjective experience
Useful - measurements, if done correctly and the subjective review

The fact is as audiophiles we've been lied to for the last 6 decades- its so prevalent that we don't even think about it. Who makes the best?: everyone, at least that is what they say. Obviously there is only one best and now many think that there is no best. Ultimately if the reviewer/manufacturer/dealer/distributor lips are moving, they are lying and about all we can do is take it home and audition it. This after 60 years- pretty sad state of affairs.

Most measurements I see ignore human hearing perceptual rules so I am of the opinion that measurements have marginal usefulness as a result. Want an example? Tell me how an amplifier sounds from its specs. Quite often there is an inverse relationship (good specs=bad sound)... whatever, If you can't tell how the product will sound from the specs then they are of little value, not unlike the Emperor’s New Clothes.

If OTOH our measurement regime used human hearing perceptual rules as the foundation, not only would we not have a lot of these endless debates but stereos would sound better too- and you would not have to audition everything to know if it would work in your system. The specs would honestly tell you what you need to know. Can't have that though: it would be the end of everything!

 

[MylesBAstor]

This is true too. And very difficult to remedy at times due to reviewer reluctance to measure things differently than other equipment or understanding the need why.

So while on the topic of the meaning of life errr... measurements, why did respondees to your recent poll choose the sound of analog over digital by a 2:1 margin? After all, there's no question that digital measures better in many, if not every important area? Must be those damn reviewers again doing sighted tests misleading everybody again.

 

[BlueFox]

So while on the topic of the meaning of life errr... measurements, why did respondees to your recent poll choose the sound of analog over digital by a 2:1 margin? After all, there's no question that digital measures better in many, if not every important area? Must be those damn reviewers again doing sighted tests misleading everybody again.

Actually, it is because analog supporters sub-consciously know digital is better, and they over compensate for their error by being overly vocal.

Measurements are nice, but an experienced ear is more relevant.

 

[MylesBAstor]

Actually, it is because analog supporters sub-consciously know digital is better, and they over compensate for their error by being overly vocal.

Measurements are nice, but an experienced ear is more relevant.

 

[Mosin]

Actually, it is because analog supporters sub-consciously know digital is better, and they over compensate for their error by being overly vocal.

Measurements are nice, but an experienced ear is more relevant.

Oh, yeah?

Well, here is the only example I can think of where I would like digital better.

 

[Groucho]

Even after all these decades of development, it's still a fair bet that 'the best' would be nothing more than zero distortion, flat frequency response, zero noise i.e. perfect linearity - and this is not a completely superfluous point to make in audiophile circles. The problem is surely that no component is ever heard in isolation, and even if it achieves exemplary performance it may be compromised by the other equipment in the chain. If objective measurements don't appear to correlate with listening (apart from the obvious sighted listening drawbacks) the reason could be simply that one component's superior performance highlights the inaccuracies in the other components. A simple example: a smallish ported speaker is going to react badly to a source with high dynamic range and frequency response all the way down. It will 'sound better' with a source that scores worse in terms of objective measurements.

So I would see this audiophile malarkey as a choice: either go for 'the best' using nothing more than objective measurements - no listening tests required - knowing that this must be applied to the whole system, and implying something like digital source, adequately-powered solid state amplification, large active speakers, no bass reflex, and knowing that this is not going to be necessarily acceptable domestically. Or decide on what is acceptable domestically and tailor its response using 'the black arts' to give the listener the right 'distortion' to make up for the lack of volume and bass. The worst of all worlds, it seems to me, is the 'black arts' version that is also domestically-unacceptable!