Why negative reviews can be unethical and misleading: 10audio.com

[Synaxis]

Most of the reviewer I've dealt with have scruples, care a lot about their work and that they do it to the best of their ability.

I'm confused.

In post #97 you stated "The problem I have is over the last 35 years I've only seen one that didn't have an ethical breach."

If I read that correctly - and I am open to the possibility I did not - your stating almost EVERY REVEIW you've read had an "ethical breech" of some sort.

That's a pretty strong statement - again assuming it means exactly as it appears.

 

[Atmasphere]

I'm confused.

In post #97 you stated "The problem I have is over the last 35 years I've only seen one that didn't have an ethical breach."

If I read that correctly - and I am open to the possibility I did not - your stating almost EVERY REVEIW you've read had an "ethical breech" of some sort.

That's a pretty strong statement - again assuming it means exactly as it appears.

Apparently I wasn't clear. What I meant by that comment was "The problem I have is over the last 35 years I've only seen one negative review that didn't have an ethical breach." I described that negative review and also its impact.

 

[Synaxis]

Apparently I wasn't clear. What I meant by that comment was "The problem I have is over the last 35 years I've only seen one negative review that didn't have an ethical breach." I described that negative review and also its impact.

OK - thank you for clarifying.

That is why I left it open that It may not have meant as typed.

 

[DaveC]

It seems engineers are far less dogmatic than audiophiles !

Some are... I think engineers are more likely to understand the inherent compromises in different designs, and how those compromises have pros and cons in different use cases. Product design requires, hopefully, a deep understanding of the compromises made by design decisions. They're also more likely to understand bias and thus see things more neutrally, and with an understanding of subjective preference. People don't value the exact same things as everyone else in life, and so there's no agreement on what the perfect speaker is, or the perfect car, or well.. anything.

Speakers are a good example because it's very easy to see the differences in speakers. Audio in general has not converged on the perfect speaker design, as everyone has different preferences that match up with the compromises made by different speaker designs. Preference for a particular speaker design is often a result of what music the listener enjoys the most. So unless you can determine what the best music is, you can't say what speaker is best, and it's pretty obvious that isn't possible.

As a result, anyone who has a dogmatic view of audio is wrong, because it's all based on subjective preference. For example, if you think balanced is always better, it's not. It's better in some use cases, but provides a solution to a problem that doesn't exist in others, in which case all it does is complicate the system for no benefit. It's often a detriment to add complexity without a reason for the complexity. This is the real truth in most home systems, there is no interference present that requires using balanced gear, a simple high-eff speaker like a single driver with one source might be best with a SET and single ended cabling. Balanced anything is a downgrade in that use case. OTOH, if you're setting up a big PA system that has tons of gear, tons of cables, many of which are often very long, and setup in a way and place that often has external interference, you need to use balanced gear to avoid problems. This is why balanced was invented and why you probably don't NEED it for a home 2-ch system, but you might need it for a big home theater with all the power and electronics setup in a large closet.

With SET amps, we all know it's favorability is entirely dependent on the speaker. With the right speaker, it may be the best choice. Other speakers will require more power and a lower output impedance to produce a superior result.

This is also the reason audio reviews are difficult and negative reviews are often not thorough enough to hold credibility. I can say from over 10 years of sending people demo cables and hopefully helping them achieve their goals, that for the reasons previously stated not everyone prefers the same thing. Despite that, there are better and worse products overall.

What I've noticed is the better I can make a cable in objective terms, the more the feedback on that product converges. Objective improvement in cables means the electrical and material properties of the cable improve, the result of this is the cable adds less of it's own character to the sound. When a cable adds its own character to the sound, it means it's not as good in some objective way, and is less neutral. The result of this is the divergence of subjective preference. in other words the reports I get back from people trying my cables differs more, if the character the cable adds to the sound is preferred people like it, if not they don't. The less character the cable adds, the more the feedback converges. I've found this equates to more predictability in recommendations and an increase in subjective preference for that cable. But there are always exceptions as some people want a particular character. In cables, this character is almost always warmth. So in some cases I do offer a warmer version of a cable... I'm starting to get away from that though, and just focus on objective superiority as I believe this will result in the best long term result. Over time preferences almost universally trend towards neutrality, meaning people value warmth less as their system improves, which makes sense as warmth covers imperfections. However, imperfections can come from the recording its self, for example a warm system is great for blasting Led Zeppelin at realistic concert SPLs. It can also come from hearing damage. So the subjective preference for warmth is NOT WRONG, it's a result of being a human with their own preferences.

Ok, I only intended to write a short reply to the quoted statement, so I'll stop here, but I hope this helps a bit with understanding the exceptional complexity of reviewing individual components used in complicated systems when combined with personal preferences. It's challenging and complex, but that's what also makes it interesting and rewarding.

 

[Jack Pot]

Some are... I think engineers are more likely to understand the inherent compromises in different designs, and how those compromises have pros and cons in different use cases. Product design requires, hopefully, a deep understanding of the compromises made by design decisions. They're also more likely to understand bias and thus see things more neutrally, and with an understanding of subjective preference. People don't value the exact same things as everyone else in life, and so there's no agreement on what the perfect speaker is, or the perfect car, or well.. anything.

Speakers are a good example because it's very easy to see the differences in speakers. Audio in general has not converged on the perfect speaker design, as everyone has different preferences that match up with the compromises made by different speaker designs. Preference for a particular speaker design is often a result of what music the listener enjoys the most. So unless you can determine what the best music is, you can't say what speaker is best, and it's pretty obvious that isn't possible.

As a result, anyone who has a dogmatic view of audio is wrong, because it's all based on subjective preference. For example, if you think balanced is always better, it's not. It's better in some use cases, but provides a solution to a problem that doesn't exist in others, in which case all it does is complicate the system for no benefit. It's often a detriment to add complexity without a reason for the complexity. This is the real truth in most home systems, there is no interference present that requires using balanced gear, a simple high-eff speaker like a single driver with one source might be best with a SET and single ended cabling. Balanced anything is a downgrade in that use case. OTOH, if you're setting up a big PA system that has tons of gear, tons of cables, many of which are often very long, and setup in a way and place that often has external interference, you need to use balanced gear to avoid problems. This is why balanced was invented and why you probably don't NEED it for a home 2-ch system, but you might need it for a big home theater with all the power and electronics setup in a large closet.

With SET amps, we all know it's favorability is entirely dependent on the speaker. With the right speaker, it may be the best choice. Other speakers will require more power and a lower output impedance to produce a superior result.

This is also the reason audio reviews are difficult and negative reviews are often not thorough enough to hold credibility. I can say from over 10 years of sending people demo cables and hopefully helping them achieve their goals, that for the reasons previously stated not everyone prefers the same thing. Despite that, there are better and worse products overall.

What I've noticed is the better I can make a cable in objective terms, the more the feedback on that product converges. Objective improvement in cables means the electrical and material properties of the cable improve, the result of this is the cable adds less of it's own character to the sound. When a cable adds its own character to the sound, it means it's not as good in some objective way, and is less neutral. The result of this is the divergence of subjective preference. in other words the reports I get back from people trying my cables differs more, if the character the cable adds to the sound is preferred people like it, if not they don't. The less character the cable adds, the more the feedback converges. I've found this equates to more predictability in recommendations and an increase in subjective preference for that cable. But there are always exceptions as some people want a particular character. In cables, this character is almost always warmth. So in some cases I do offer a warmer version of a cable... I'm starting to get away from that though, and just focus on objective superiority as I believe this will result in the best long term result. Over time preferences almost universally trend towards neutrality, meaning people value warmth less as their system improves, which makes sense as warmth covers imperfections. However, imperfections can come from the recording its self, for example a warm system is great for blasting Led Zeppelin at realistic concert SPLs. It can also come from hearing damage. So the subjective preference for warmth is NOT WRONG, it's a result of being a human with their own preferences.

Ok, I only intended to write a short reply to the quoted statement, so I'll stop here, but I hope this helps a bit with understanding the exceptional complexity of reviewing individual components used in complicated systems when combined with personal preferences. It's challenging and complex, but that's what also makes it interesting and rewarding.

I cannot agree more. I would like to add that, as consumers, we should applaud and celebrate the wonderful diversity of Hifi "perspectives". Such diversity of choice would not exist without the plethora of manufacturers who investigate sound reproduction. And at the risk of being ostracized, I consider the cable loom as one of the more important components of a hifi-system. Invest in a good cable loom and it will survive many other upgrades and consistently bring out the best of purchased equipment. Good power management, good equipment supports, good cable loom, and where necessary, good room treatment constitute the infrastructure of a hifi-system on which ALL else depends. Spend wisely, invest in boring and mostly "invisible" infrastructure and save fortunes on pointless other equipment upgrades.

 

[Tangram]

In my experience, any overtly negative review should be read with caution simply because it’s a significant outlier, in the order of two standard deviations. A positive review with no nitpicks (far more common) should also be read with caution. How many times have we read speaker reviews where there’s a clear mismatch with the amp (because that’s all the reviewer had on hand) yet you’d never know it from the review? Or a reviewer living in a tiny New York apartment ascribes sonic attributes to a speaker that would be challenging to achieve in a much better room? For me, the best reviews give the bad with the good, but they give reasons why another person may get a different outcome. I also care about quality and ease of use, and enjoy truly insightful design commentary if a novel approach is used.

For me, as a customer, any extreme review, negative OR positive, is taken with a grain of salt. It’s actually pretty easy for me to be objective. After all, my heart and soul haven’t been poured into a project over many years, the success or failure of which has real implications for those involved in its design, production, marketing and sales. So I truly empathize when a company feels slighted by a negative review. But rest assured, most end users are astute enough to recognize that rare hit piece when they see it.

 

[mtemur]

Most of the audio component reviews (I’m not saying all, but most) are there to convince readers that it justifies its price.

A review always has a conclusion like “it’s not the best but well deserves its price tag” or “punches way above its league”. This way, the reviewer plays safe and avoids clear statements about whether the component under review really sounds good or bad and tries to convince you how logical and rational it is to spend money on it.

It’s a win-win situation for both the manufacturer and the magazine. More importantly, you as a reader, cannot claim the reviewer is misleading because nobody can prove it doesn’t punch above its league or what the actual league it’s on.

 

[PeterA]

Most of the audio component reviews (I’m not saying all, but most) are there to convince readers that it justifies its price.

A review always has a conclusion like “it’s not the best but well deserves its price tag” or “punches way above its league”. This way, the reviewer plays safe and avoids clear statements about whether the component under review really sounds good or bad and tries to convince you how logical and rational it is to spend money on it.

It’s a win-win situation for both the manufacturer and the magazine. More importantly, you as a reader, cannot claim the reviewer is misleading because nobody can prove it doesn’t punch above its league or what the actual league it’s on.

Very true. I like reviews that tell me something about the actual company and component and then describes what the reviewer actually hears from his system with the component while listening to specific music recordings, and then ideally, how what he hears may differ from another component he has on hand.

 

[tima]

Most of the audio component reviews (I’m not saying all, but most) are there to convince readers that it justifies its price.

A review always has a conclusion like “it’s not the best but well deserves its price tag” or “punches way above its league”. This way, the reviewer plays safe and avoids clear statements about whether the component under review really sounds good or bad and tries to convince you how logical and rational it is to spend money on it.

It’s a win-win situation for both the manufacturer and the magazine. More importantly, you as a reader, cannot claim the reviewer is misleading because nobody can prove it doesn’t punch above its league or what the actual league it’s on.

Interesting -- I think this is the case some of the time though I'm sceptical of 'most'. Your comments could be compelling if you included links to reviews from well known reviewers that are examples. Should be easy to find.

I'll list msrp but almost never talk about price or value because I typically don't consider those factors in gauging a component. Can't recall if I've used language similar to what you cite.

What do people look for or want to read in a review's conclusion?

 

[PeterA]

What do people look for or want to read in a review's conclusion?

A short summary of the main characteristics of the component’s sound, and a very clear expression of whether or not the reviewer likes the sound of the component.

 

[mtemur]

What do people look for or want to read in a review's conclusion?

That’s a fair question. I don’t know what others want, but here’s what I want to read:

• How do the instruments sound? How are the dynamics? Is it lifelike?

Instead of vague descriptions like “blacker backgrounds,” “deep soundstage,” or “quietness” tell me how the piano sounds? Does the kick drum really sound something is kicking under the couch? Soundstage is more related to the room than the equipment itself, yet reviews still mention it. Additionally, nobody knows if a deep soundstage is inherently good or bad. If it’s a multi-miked recording (which almost all recordings are, except 1950s–60s classical), where is that soundstage even coming from? Similarly, how black is “black enough” for a blacker background?

• Regarding comparisons to other equipment, I want to know which one the reviewer prefers to keep listening to. There’s often a mixed comparison, like, “This one is better at X, and the other is better at Y.” Just tell us—what do you prefer? Which one is better overall? When comparing similarly priced components, you almost always end up preferring one over the others. One wins, even if it doesn’t win in every single area.

Bottom line; most reviews are very long, fun to read but hardly say if the component under review is any good. Vague descriptions and comparisons left you confused without a clear answer to that question. In my opinion only one thing is clear with those reviews; if you have the money you should buy this.

BTW I don’t think your reviews are like the ones I’m complaining.

 

[PeterA]

That’s a fair question. I don’t know what others want, but here’s what I want to read:

• How do the instruments sound? How are the dynamics? Is it lifelike?

Instead of vague descriptions like “blacker backgrounds,” “deep soundstage,” or “quietness” tell me how the piano sounds? Does the kick drum really sound something is kicking under the couch? Soundstage is more related to the room than the equipment itself, yet reviews still mention it. Additionally, nobody knows if a deep soundstage is inherently good or bad. If it’s a multi-miked recording (which almost all recordings are, except 1950s–60s classical), where is that soundstage even coming from? Similarly, how black is “black enough” for a blacker background?

• Regarding comparisons to other equipment, I want to know which one the reviewer prefers to keep listening to. There’s often a mixed comparison, like, “This one is better at X, and the other is better at Y.” Just tell us—what do you prefer? Which one is better overall? When comparing similarly priced components, you almost always end up preferring one over the others. One wins, even if it doesn’t win in every single area.

Bottom line; most reviews are very long, fun to read but hardly say if the component under review is any good. Vague descriptions and comparisons left you confused without a clear answer to that question. In my opinion only one thing is clear with those reviews; if you have the money you should buy this.

BTW I don’t think your reviews are like the ones I’m complaining.

Excellent post. I agree with all of your points about reviews. I think Tim was asking specifically about the conclusion at the end of the review. I also agree with you that Tim's reviews are excellent and do in fact describe what he hears when listening to specific recordings. He goes into good detail about the music and sound. And you do not read or hear about him going to lavish dinners at the shows or with the manufacturers. There are some very good reviewers out there. At this point, I rarely read reviews. If someone sends me a link, I read to learn more about the music than I do the component, as my system is basically done.

 

[F208Frank]

Recently there was a 'review' of our class D amps on 10audio.com.

The amps didn't come from us. The author claims he bought them. If he did, it was because he knew very well that we would never release a demo sample to him, as in the past he had done unethical things in his reviews.

This blew up recently when this video appeared on youtube, along with this thread on audioshark.com.

We don't even know if Jerry (the owner of 10Audio) really had a set of our class Ds on hand. Yes, IMO he's dishonest enough that I wouldn't put it past him to fake it. But giving him the benefit of the doubt, we don't know what condition the amps were in, if they had been modified by someone and so on. Heck, they might have been set up for 235V operation (such as set of amps will run on 117V but won't make power).

Now this all started about 25 years ago, back when we still offered our M-60 amplifier as a kit. Jerry approached us about buying one, which he successfully built and liked. He wanted to do a reveiw of it for his new website. He liked it so much that he later bought from us an MP-3 preamp. He reviewed and liked that too.

But then apparently he wondered, since we offer the M-60 with a number of options (Caddock resistors, Teflon coupling caps and a power supply boost option which doubles the capacity in the output section power supply) what a fully optioned set of M-60s might sound like. Rather than talk to us about it, he bought a used set he found and reviewed them. He didn't like them. Turned out the set he bought used was a kit, which was not mentioned in the review, and a poorly built one at that, barely running. He sold them to a customer of ours in New York City (who is still a customer of ours, named Jeffery W.). Jeffery contacted us about getting the amps fixed. It was about that time we found out about the negative review.

The amps turned out to be a rat's nest inside with bad and missed solder joints. We cleaned them up as best we could and returned it to Jeffery, who said a few months back he still has them.

Then I confronted Jerry about this issue. That didn't go well- he was unable to admit to his ethical gaff. Instead he changed the existing reviews of the M-60 and MP-3 from good to bad as retaliation. When we discovered that we posted what we knew on audioaylum.com. The resulting kerfuffle resulted in Jerry not posting on audioasylum for a few years.

There is an internet archive called the Wayback Machine. It archives everything on the searchable web.

Below you can see a couple of links of reviews of the MP-3 preamp. Scroll down to the bottom and compare what you see between the two pages. On any normal review site these two pages would be unchanged or at least all the text would be the same. But in this case you can see additions, and a change a rating in the later page. The wayback machine does not have a snapshot from Novemeber 2002 so we can't see what the original review was. But we can still see that he was changing the existing content anyway:

http://web.archive.org/web/20040111140344/http://www.10audio.com/AS_MP3.htm
http://web.archive.org/web/20070524082912/http://www.10audio.com/as_mp3.htm

Why Jerry chose this time to open up this old wound is beyond me; one would think it best to let sleeping dogs lie.

Mike, the Greek Audio Geek on YT, didn't know anything about this. I'm not a content producer so I don't know what sort of pressures exist, but IMO he wasn't making the best move in his video I linked above.

Here is my point: when you see a bad review, in my opinion/in my experience, its unethical. The reason is simple: if the equipment is really bad the best thing to do as a review is to simply send it back and not mention it publicly.

Bad reviews occur because equipment might be damaged (or in the case of this post, might not even exist) in shipment, the reviewer might have troubles setting the equipment up, there could be a system incompatibility, the reviewer (as we see with Jerry) might have an ax to grind, on in some cases the organization for whom the reviewer works might have an agenda- like a punishing review if no ad campaign occurs. I've been in this industry long enough I've seen all of these things go down.

For example Gryphon got a bad review in The Absolute Sound decades ago. It literally ended their US distribution. I was in the room at CES and witnessed the reviewer threatening the owner of Gryphon that if he didn't just give him the equipment that was the subject of the review that it would be a bad one. I don't think TAS had any idea about this; it was unethical behavior on the part of the reviewer. He fell into the category of reason I mentioned above as 'had an ax to grind'; he was also trying to be on the take. Harvey Rosenburg did that to us decades ago as well. In that case we found out he had sold a review sample of ours when the guy he sold the amps to contacted us for a user manual. When we approached Harvey about paying us the industry accommodation, he declared war on us. I found out later he had done this to a number of other companies (one who was forced to fold due to the financial loss) who were smarter than me in that they decided to just deal with the loss rather than Harvey badmouthing them in the press.

I can go on put you get the point. Always be suspicious of a bad review.

Thanks for sharing. Sorry you had to go through that.