How do we improve the reviewing process of High End equipment?

FrantzM · Jul 30, 2010

Hi

This was prompted by a very judicious remark from Kal Rubinson in another thread:

Kal Rubinson said:
I believe this is somewhat backwards. Given the almost infinite human capacity for bias and self-delusion, wouldn't it be better if the procedure began with extended listening (and note-taking), followed by careful measurements and then, only at the end, another period of listening?

Whatever is said about the reviewers, reviews are an important part of most human activities. It is physically impossible for any of us to have access to all the gear that are out there.. Even less the time to audition these.
That would be the role of reviewer. Ideally an objective assessment of his/her encounter with a component. How it sounded to him/her at the time of review and under which conditions. It would be interesting that they measure and try to correlate these with their subjective impressions.

What else would you people like to see in High End Audio Reviews? How do you think the process can be made better?

I for one would like the reviewer to push aside his or her biases and to try once in a while some blind listening , not DBT simply remove at time the knowledge or what is being played ... his certainly invoke some level of difficulty but ... I am wishing..

I also would like to see reviewers not falling or the "price point" notion.. One which has helped make the high end audio akin to luxury gear practices : High Price ... no correlation with performance ... or to be charitable small increments in performance ... IMO opinion the reviewers bear some, certainly not all, responsibility to this price flight toward the stratosphere ...

Let's all make sure the discussion remains polite and civil. This thread is not mean to incite anyone to throw bullets and daggers at reviewers in general or in particular those reviewers who are also members of this forum.... The idea is to be constructive.

Kal Rubinson · Jul 30, 2010

FrantzM said:
I for one would like the reviewer to push aside his or her biases and to try once in a while some blind listening , not DBT simply remove at time the knowledge or what is being played ... his certainly invoke some level of difficulty but ... I am wishing.

This is difficult for many reasons but two of them are that most reviewers operate solo without any support to implement a blind (my wife sometimes helps) and that the most useful auditions are long term requiring massive effort to maintain the blind (no peeking behind the rack). The best I have done, so far, was in my efforts in comparing the Oppo BDP-83 and the 83SE. http://www.stereophile.com/budgetcomponents/music_in_the_round_41/index.html

Kal Rubinson · Jul 30, 2010

tomelex said:
Say mid level audio phile gear. $500 cd players, $1500-2000 speakers, $500 amps. (forget TT to keep example simple). Collect 5 or 10 of the best in each class. Put them in a totally dark room, and assemble 5 or so reviewers. Each reviewer must have his audiogram published from the current year, so we may compare to our hearing response atleast somewhat, ie if you are over 50, you are not hearing what a 20 year old is hearing. Acoustic instrument musicians must play through and without amplification as part of the tests, and this includes also bass and drum and say violin and maybe clarionet or whatever needed to capture the range and timbre.

Keep playing and voting on the combinations until there is a rank ordering. This may take a month to do. Then, publish this list and now, there is some rational basis for selection of these components. That to me would provide real value instead of just pure entertainment.

If you are able to set up such a situation, why not just let the subscribers (who will be paying a huge fee for this regardless of whether it is published or not) sit in and decide for themselves? Of course, things might sound a bit difference in any given individual's room but this eliminates a significant variable: the reviewer!

MylesBAstor · Jul 30, 2010

Kal Rubinson said:
If you are able to set up such a situation, why not just let the subscribers (who will be paying a huge fee for this regardless of whether it is published or not) sit in and decide for themselves? Of course, things might sound a bit difference in any given individual's room but this eliminates a significant variable: the reviewer!

Maybe reviewers could be paid more so it could be full time gig. There are only few reviewers that I know of that received a salary and aren't paid by the piece (review that is, not the equipment

)!

Kal Rubinson · Jul 30, 2010

MylesBAstor said:
Maybe reviewers could be paid more so it could be full time gig. There are only few reviewers that I know of that received a salary and aren't paid by the piece (review that is, not the equipment )!

Sure but do you really think that is economically feasible for the publication? You should know. ;-)

Kal

MylesBAstor · Jul 30, 2010

Kal Rubinson said:
Sure but do you really think that is economically feasible for the publication? You should know. ;-)

Kal

Oh of couse not

But one can dream. Well...maybe that's close to hallucinate

Johnny Vinyl · Jul 30, 2010

Although I don't subscribe to any of them anymore, I do buy several from our local Chapters Bookstore on a monthly basis.

The one issue I have, and it's not really an issue (cause I don't subscribe), is that the vast majority of people reading them can't afford 50% or more of the products that are being reviewed. Why do I want to read a review on the latest Burmeister CDP that costs more than the average basement renovation? Is it interesting to read? Yeah it can be, but it's not relevant to me and many others. I wish they would cater more to the JSP audiophile as opposed to the few that can actually afford the products being reviewed.

John

MylesBAstor · Jul 30, 2010

John72953 said:
Although I don't subscribe to any of them anymore, I do buy several from our local Chapters Bookstore on a monthly basis.

The one issue I have, and it's not really an issue (cause I don't subscribe), is that the vast majority of people reading them can't afford 50% or more of the products that are being reviewed. Why do I want to read a review on the latest Burmeister CDP that costs more than the average basement renovation? Is it interesting to read? Yeah it can be, but it's not relevant to me and many others. I wish they would cater more to the JSP audiophile as opposed to the few that can actually afford the products being reviewed.

John

Yes but how many read Watchtime? How many people can afford a repeater? Or looking at Ferraris etc? I think that many like to dream and know about the best. Others just are jealous and trash the gear and audio.

Steve Williams · Jul 30, 2010

Ill bite. Whats a JSP audiophile

Johnny Vinyl · Jul 30, 2010

Steve Williams said:
Ill bite. Whats a JSP audiophile

Joe Six Pack

Johnny Vinyl · Jul 30, 2010

MylesBAstor said:
Yes but how many read Watchtime? How many people can afford a repeater? Or looking at Ferraris etc? I think that many like to dream and know about the best. Others just are jealous and trash the gear and audio.

I'll agree ...it is nice to dream and I do it all the time(LOL), but it leaves me in a vacuum. I buy mags to learn from and look to them as maybe giving me a direction as to what is relevant to my situation. I am not knocking the major mags by any stretch, but is the "dream" their only raison d'etre? If it is, I'll defer to my collegues here and elsewhere for advise, because I'm not really getting in print.

John

MylesBAstor · Jul 30, 2010

John72953 said:
Joe Six Pack

Be careful..didn't Sarah Palin trademark that phrase?

caesar · Jul 30, 2010

FrantzM said:
Hi

What else would you people like to see in High End Audio Reviews? How do you think the process can be made better?

I for one would like the reviewer to push aside his or her biases and to try once in a while some blind listening , not DBT simply remove at time the knowledge or what is being played ... his certainly invoke some level of difficulty but ... I am wishing..

I also would like to see reviewers not falling or the "price point" notion.. One which has helped make the high end audio akin to luxury gear practices : High Price ... no correlation with performance ... or to be charitable small increments in performance ... IMO opinion the reviewers bear some, certainly not all, responsibility to this price flight toward the stratosphere ...

.... The idea is to be constructive.

Ok here are a few ideas:

- publish pictures of the room
- post/ publish preferred sonic signature
- 90 day equipment loan limit
- in addition to the present reference system, publish/ post previous reference systems. Explain why changes were made
- disclose whether each piece of gear was purchased or is on loan
- pool units at the same price point and send them from reviewer to reviewer for multiple perspectives/ compares contrasts

Of course this is just a dream. This is kind of like politicians in Washington or sharks on Wall Street watching over themselves.

MylesBAstor · Jul 30, 2010

caesar said:
Ok here are a few ideas:

- publish pictures of the room
- post/ publish preferred sonic signature
- 90 day equipment loan limit
- in addition to the present reference system, publish/ post previous reference systems. Explain why changes were made
- disclose whether each piece of gear was purchased or is on loan
- pool units at the same price point and send them from reviewer to reviewer for multiple perspectives/ compares contrast

Of course this is just a dream. This is kind of like politicians in Washington or sharks on Wall Street watching over themselves.

The problem with these requests and print publications is the space required; that's not an issue with online mags.

Actually several online mags already do this. Positive Feedback for one does 1, 2, 5, and 6. Three is somewhat unrealistic for a multitude of reasons. Six is also very difficult to do given the number of mags asking for review samples.

Phelonious Ponk · Jul 31, 2010

MylesBAstor said:
The problem with these requests and print publications is the space required; that's not an issue with online mags.

Actually several online mags already do this. Positive Feedback for one does 1, 2, 5, and 6. Three is somewhat unrealistic for a multitude of reasons. Six is also very difficult to do given the number of mags asking for review samples.

Comprehensive measurements, blind listening conditions, panel reviews...any/all of these would serve readers/consumers much better than the current, utterly subjective system. I understand that the current system is dominated by individual freelance writers without the resources for a, the proper conditions for b or the proximity for c, but we're dreaming, right? And without any of the above, a review is one man's opinion with all of his personal prejudices and expectations in play. Probably only useful if you've read many of that one man's reviews and know his mind well. I don't expect them to be useful. I read them for their entertainment value only and often find the most outrageous and least credible of the lot to be the most entertaining. I won't provide links.

P

MylesBAstor · Jul 31, 2010

Phelonious Ponk said:
Comprehensive measurements, blind listening conditions, panel reviews...any/all of these would serve readers/consumers much better than the current, utterly subjective system. I understand that the current system is dominated by individual freelance writers without the resources for a, the proper conditions for b or the proximity for c, but we're dreaming, right? And without any of the above, a review is one man's opinion with all of his personal prejudices and expectations in play. Probably only useful if you've read many of that one man's reviews and know his mind well. I don't expect them to be useful. I read them for their entertainment value only and often find the most outrageous and least credible of the lot to be the most entertaining. I won't provide links.

P

There are magazines for you. OH, they're out of business? Too bad.

And you know for all your blustering about bias, DBT etc, panel reviews, give me five examples of reviews where the subjective review totally disagreed with what the measurements found. A couple will always sneak through no matter what the science (even in peer reviewed academic journals there are retractions). That's life -- we [human beings], well maybe you are -- aren't perfect. The only review I can remember in Stereophile was an old Fremer review where he thought the gear sounded better than the measurements might indicate. OTOH, I remember numerous instances of where it was pointed out in reviews how what the reviewer commented on was backed up by the measurements. I don't know about you but I'd take 99.9% accuracy.

Face it. The ear is still the best instrument God ever built for hearing. Can it be fooled? Of course. And so can any of our other senses. But does that make it any worse than an instrument that can only tell you what its asked and is ALWAYS subject to experimental methodology and misinterpretation? And might I add, til the new new BUZZ measurement comes out???? And why do we have new measurements? Because the old ones didn't work!!!

OTOH, there are far more examples of where measurements led to erroneous conclusions. Or my PhD advisor would have said, experiments led us down the garden path meaning if curing mice of cancer was our goal, we would have reached it 25 years ago.

Phelonious Ponk · Jul 31, 2010

Relax, Miles. Nobody said anything about DBT. And I believe I made it clear that I understand that freelance reviewers do not have the resources to run measurements. Casual, blind listening, however, is not that difficult to accomplish, with the exception of speakers. And can you honestly tell me that you can give a more objective evaluation of a component because you know when it is and is not in the signal chain? That your evaluation would be less objective without that knowledge? And where's the "bluster?" I read reviews for entertainment value only. Others, as I pointed out, may read particular reviewers religiously, understand their positions and their points of view and actually gain useful perspective from their opinions. But as long as it is a purely subjective system, it seems to me that any real knowledge gained by the reader is going to require serious knowledge of the writer. have I got that wrong?

And are the measurements we have ineffective, or are reviewers subjective judgements often backed up by the measurements we have? If it's both, it speaks well of neither, but I suspect that's not quite what you meant to say.

P

caesar · Jul 31, 2010

MylesBAstor said:
Six is also very difficult to do given the number of mags asking for review samples.

True - I agree. However, this is where the real value is for the customer! I believe you have those new CJ amps. I am sure they are great - I don't need to read the review to guess that. If you compare them to a $10 K amp, as most reviews would, what value does that really add?

But let's say you also had the new BAT Rex, VTL, and ARC tube amps and compared and contrasted these with several different kinds of speaker loads. Then you sent them on to your colleagues for their impressions with several other speakers they enjoy. This is where the value to the customer would really be. The car magazines manage to do this. Why not in audio?

Kal Rubinson · Jul 31, 2010

caesar said:
The car magazines manage to do this. Why not in audio?

The cars can be driven from place to place. ;-)

MylesBAstor · Jul 31, 2010

caesar said:
True - I agree. However, this is where the real value is for the customer! I believe you have those new CJ amps. I am sure they are great - I don't need to read the review to guess that. If you compare them to a $10 K amp, as most reviews would, what value does that really add?

But let's say you also had the new BAT Rex, VTL, and ARC tube amps and compared and contrasted these with several different kinds of speaker loads. Then you sent them on to your colleagues for their impressions with several other speakers they enjoy. This is where the value to the customer would really be. The car magazines manage to do this. Why not in audio?

Caeser:

I don't disagree with anything you said. I think the best days of TAS were when HP did shoot outs among different amps, cartridges, etc.

I'm also going to add something that Albert Porter and I were discussing last night-that is at the upper echelon it still comes down to personal preference. No one piece of equipment is going to do it all --and the consumers final choice often comes down to what the user values in musical reproduction. What a good reviewer should be able to do is convey the strengths and weaknesses of a component - and that is made easier in a system that is low in coloration.

But as far as having something like the ART as well as say an ARC 610T, etc., it also serves as a reference pt. to see what is achievable in amplifier design. It's not as good as live music. but it kinda serves a similar purpose.

How do we improve the reviewing process of High End equipment?

Member Sponsor & WBF Founding Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Member Sponsor & WBF Founding Member

Well-Known Member

Site Founder, Site Owner, Administrator

Member Sponsor & WBF Founding Member

Member Sponsor & WBF Founding Member

Well-Known Member

Well-Known Member

Well-Known Member

New Member

Well-Known Member

New Member

Well-Known Member

Well-Known Member

Well-Known Member

Similar threads