Changing variables in a review

PeterA

Well-Known Member
Dec 7, 2011
9,496
6,376
1,335
North Shore of Boston
This topic came up in Marty’s amplifier thread. Member cableman questioned Roy Gregory’s review methods. Mr. Gregory stated that he will change speaker positioning within the room to better optimize the sound of various amplifiers being reviewed. My question is: does changing more than one variable at a time change your opinion about the value of the review of the component being compared?

What would you think of a turntable review if the reviewer changed the tonearm and cartridge to optimize the sound of the turntable in a given system context?

This is my response to Ron Resnick in the thread which I deleted because it is off-topic to Marty’s thread.

After exhaustive research I have not found this in the Ten Commandments. You state this as though it were a law of nature or a maxim of science. It is neither; it is merely your opinion..

You have your opinion as to how amplifiers should be evaluated. Repositioning the loudspeakers to maximize the subjective sound quality of the combination is an alternative review protocol.

I would think that the fewer variables in a review, the better. But I have no dog in this hunt.

I think either amplifier review protocol is rational and defensible. All that really matters, I think, is that a reviewer follows his/her selected protocol consistently.

Ron, the reader is free to believe whatever he wants from a review. Personally, I don’t want to see the reviewer change more than one variable at a time before describing the effect on the sound. The exception I can think of is replacing an entire chain of electronics from one brand to another brand.

I did not know that this was a consistent process for Roy Gregory. I appreciate that cableman pointing it out. Knowing this about Mr. Gregory‘s amplifier reviews has changed my opinion about the validity of the reviews.

Marty did not do that when he made the comparison. I did not do it when I compared two different electronic chains in my system. Knowing the review process is helpful because it helps us place value on the results of the review.
 
  • Like
Reactions: Cableman

Mike Lavigne

Member Sponsor & WBF Founding Member
Apr 25, 2010
10,450
6,153
1,420
how long is a piece of string?

how can you know what you don't know?

you have to test things, or accept that you did not. what is relevant and what is not relevant? if you assume that a speaker<->room context will not affect how an amp performs then that is neither right nor wrong. but it's an assumption you made. my son-in-law the laboratory Phd in physics is a very humble guy who always is focused on what he does not know, not what he thinks he knows. good advice.

if i was a reviewer (or a dealer commonly changing speakers and amps) and i had a room and resident speakers, that allowed me to easily test this, then i would have an opinion about it. how many reviewers are in such a position to be able to test this regularly (or are sufficiently rigorous)?

i would agree that i would not expect speaker positioning to change my viewpoint of amplifiers. but i would admit to never considering it and if someone claimed that, i would need to be open minded about it. it certainly would not cause me to think less of the reviewer.....more the opposite. i would view their amplifier viewpoints as more thoroughly investigated.
 
Last edited:

PeterA

Well-Known Member
Dec 7, 2011
9,496
6,376
1,335
North Shore of Boston
Mike, I think a reviewer can do whatever he wants as long as he explains in the published review how he set up the conditions. Ideally, I would like a straight amplifier comparison first without other variables changing, and then if the reviewer finds he can improve sonics of the overall system by moving around the speakers, then that is fine and it should be explained and will become an interesting part of the review.

Would changing around power cords and finding the best sounding amp stand or platform affect one’s opinion about the review of an amplifier?

I think different reviewers approach their review parameters differently. As a reader, I would always question multiple variables being changed in the comparison unless there was very careful corresponding descriptions with each change.
 

DonH50

Member Sponsor & WBF Technical Expert
Jun 23, 2010
3,860
225
460
Monument, CO
If the amplifier interacts with the speakers in such a way as to change the frequency response (or other things) then changing the speaker and/or listening position can negate or emphasize the interaction. For example, the amplifier's output impedance changes over frequency, and thus can change how a given speaker sounds with that amplifier. Moving the speakers may counter, or enhance, the change the amplifier makes in (to) the system (speakers). In that case a couple of things come to my mind:

1. It is very important that the reviewer delineate, or tell us, exactly what changed and why to help isolate the character of the amplifier from the speakers.

2. It is very important to know about the speakers, room, and reviewer's preferences since it is unlikely I have the same room, speakers and taste as the reviewer. By analyzing the amplifier and speaker parameters hopefully I can decide what is happening and decide if the amp will do the same (or not) in my system.

The latter may require some technical acumen to understand the interaction, and/or lots of trial and error with the amp in house. Did the bass become less obvious, perhaps due to lower output impedance (higher damping factor, better control of the speaker)? Or more "boomy" due to higher output impedance so speaker peaks in the bass are emphasized? Did the treble rise or fall, perhaps due to the falling impedance of an ESL? Etc.

Personally I would not change another variable in a review but understand the desire to make the amplifier sound its best. The fear is that it may be much harder for me to do the same in my system, and thus difficult to judge from the review if the amp is a good fit for me or not.

IMO - Don

Edit: @PeterA was writing whilst I was typing; looks like we have similar viewpoints.
 

bonzo75

Member Sponsor
Feb 26, 2014
17,914
7,684
1,415
London

Mike Lavigne

Member Sponsor & WBF Founding Member
Apr 25, 2010
10,450
6,153
1,420
since we cannot duplicate the reviewer room or set-up exactly, i don't need chapter and verse about every step he took. but the more he shares the better too. i'm wanting to know his thoughts after he has optimized the context.

tube rolling and power cord or interconnect changing, which we as end users can duplicate, are relevant things we want to know. and maybe footers and such too other than stock.

if the review is just about a compare with another product, then yes, speaker position or adjustment changes would be relevant info. but it's a judgment call by the reviewer as to how detailed he gets about his optimization process.
 
  • Like
Reactions: treitz3 and Lee

ddk

Industry Expert
May 19, 2013
5,835
3,150
870
Utah
This topic came up in Marty’s amplifier thread. Member cableman questioned Roy Gregory’s review methods. Mr. Gregory stated that he will change speaker positioning within the room to better optimize the sound of various amplifiers being reviewed. My question is: does changing more than one variable at a time change your opinion about the value of the review of the component being compared?

What would you think of a turntable review if the reviewer changed the tonearm and cartridge to optimize the sound of the turntable in a given system context?

This is my response to Ron Resnick in the thread which I deleted because it is off-topic to Marty’s thread.



Ron, the reader is free to believe whatever he wants from a review. Personally, I don’t want to see the reviewer change more than one variable at a time before describing the effect on the sound. The exception I can think of is replacing an entire chain of electronics from one brand to another brand.

I did not know that this was a consistent process for Roy Gregory. I appreciate that cableman pointing it out. Knowing this about Mr. Gregory‘s amplifier reviews has changed my opinion about the validity of the reviews.

Marty did not do that when he made the comparison. I did not do it when I compared two different electronic chains in my system. Knowing the review process is helpful because it helps us place value on the results of the review.
Honestly I don't understand why @RoyGregory 's comment is creating this much controversy. I can't speak to what went on at @marty 's but IME introducing a new component requiring a change in setup isn't unusual; either because of interaction or just highlighting what is off with the setup. In that exchange @RoyGregory had a very different experience from @marty and offered a plausible reason, what's the issue?

Who wrote it in stone that you can change only one variable at a time? If something doesn't initially work out doesn't mean that it's bad why not try a few things and see before deciding? Even if speakers are already setup well very small changes can make huge differences.

david
 

marty

Well-Known Member
Apr 20, 2010
2,449
2,581
670
United States
Who wrote it in stone that you can change only one variable at a time?
Of course you can. It's not the way I would approach an experiment from a scientific perspective, but nobody says you can't do it. Knock yourself out! Do whatever you think will get you to the promised land. As far as believing that an amplifier will change the acoustic properties of room resonance is concerned (and hence, speaker placement), I'll just have to respectfully disagree. Tell that to acousticians and watch them laugh at you.
 

sbnx

Well-Known Member
Mar 28, 2017
636
478
155
I would like to pose this question. How does someone review speakers? The new pair of speakers is always setup in a slightly different location to the original pair. If we were to hold to the "one variable at a time" philosophy then the new speakers should be put in EXACTLY the same place as the pair that just left. We all know this doesn't work and fully accept that the reviewer had to reposition the new speakers to get good sound in his/her room. Yet, when Roy mentioned that he repositions the speakers when he gets a new amp to achieve best sound everyone (well not quite everyone) freaked out.

The speakers interact with the room exciting some frequencies and not others. The woofer size, port (front, back or sealed), and even the location of the woofer dictate how that speaker is going to interact with the room. This would necessitate a positional adjustment for best sound compared to the original speaker. The amplifier interacts with the speaker. Lower the output impedance will result in greater woofer control and huge power supplies can push the woofer harder (supply more current). If this is the case then why is it unreasonable to expect that one might need to move the speaker a little to achieve best sound with the new amplifier.

IMO the one variable method is what leads to the merry-go-round and spending a lot of money unnecessarily in search for the perfect match. If we put the speakers down and fix them to never move again then we are simply seeking equipment to change the sound. This is like the tail wagging the dog. I say this as nothing has as profound an impact on what we hear as speaker position (given that the best listening position has already been found).
 

docvale

Well-Known Member
Mar 22, 2011
531
48
380
Briarcliff Manor, NY
Giving my opinion here, not as an experienced audiophile (I am not), but as an experienced biomedical research scientist.

if you want reviews in which a scientific principle is applied, you need:
- to formulate an hypothesis (does this amplifier perform better than my reference?)
- define a meticulous experimental setting (identify all the potential variables, maintaining the comparison groups as controlled as possible)
- run the experiment collecting quantitative data
- verify the data are reproducible
- verify the hypothesis (typically, running all the possible experiments that would invalidate the hypothesis)

what I wrote above is not my opinion, but it is the scientific method.
if you want audio reviews with scientific method, you need to go through measurements then.

I don’t want to go to the measurements vs humanistic reviews.
My opinion is that an audio review, to be useful to me, does not necessarily have to say whether a device that is under scrutiny is better, equal or worse than a given reference. Mostly because the chances I’m familiar with the specific system of the reviewers are null.
I would instead second the distribution of all the potential elements that were put in place to get the best of a given device, just to understand whether I would be able, purchasing that given component, to squeeze the best of it.
Obviously, this preference becomes strong if I’m reading a review to get informed about a specific purchase I am planning to do. In that case, my approach is to read as many reviews I can (somehow, this is scientific method as I try to check if certain “findings” can be reproduced).
Typically, I read reviews for entertainment purposes, so I’d say I would be quite flexible in terms of tolerance towards reviewing disclosures.
 
  • Like
Reactions: stirlingtrayle

ddk

Industry Expert
May 19, 2013
5,835
3,150
870
Utah
Of course you can. It's not the way I would approach an experiment from a scientific perspective, but nobody says you can't do it. Knock yourself out! Do whatever you think will get you to the promised land. As far as believing that an amplifier will change the acoustic properties of room resonance is concerned (and hence, speaker placement), I'll just have to respectfully disagree. Tell that to acousticians and watch them laugh at you.
As I mentioned I can't speak to your experience Marty in this case just my own. Who said the amp affects room resonance I was only speaking about it's relationship to the speaker, things change specially at frequency extremes. Specially if the speakers are in an ideal location an inch or two in any direction can have significant impact on the sound. Personally I don't see trying out gear the same as a controlled science experiment, there are always variables and I see the boxes only as tools to achieve a goal, nothing more.

david
 

PeterA

Well-Known Member
Dec 7, 2011
9,496
6,376
1,335
North Shore of Boston
Honestly I don't understand why @RoyGregory 's comment is creating this much controversy. I can't speak to what went on at @marty 's but IME introducing a new component requiring a change in setup isn't unusual; either because of interaction or just highlighting what is off with the setup. In that exchange @RoyGregory had a very different experience from @marty and offered a plausible reason, what's the issue?

Who wrote it in stone that you can change only one variable at a time? If something doesn't initially work out doesn't mean that it's bad why not try a few things and see before deciding? Even if speakers are already setup well very small changes can make huge differences.

david

I don't think there is a lot of controversy David. I think it is all fine if the reviewer discloses the process and comments along the way as he makes changes. I would prefer if the reviewer's starting point is a simple insertion of one component for another. Going on later to optimize a particular set up, or fine tune it, or however you want to describe the process is fine, and it could become an interesting part of the published review.

My only point is that the reader should know what the variables are under which the component is reviewed, and whether or not more than one has been changes. I think it changes the outcome of the comparison. If footers, power cords, and/or speaker position are changed without describing the full sonic results, the reviewer is doing a disservice to his readers, and the review, for me, has less value.
 

ddk

Industry Expert
May 19, 2013
5,835
3,150
870
Utah
I don't think there is a lot of controversy David. I think it is all fine if the reviewer discloses the process and comments along the way as he makes changes. I would prefer if the reviewer's starting point is a simple insertion of one component for another. Going on later to optimize a particular set up, or fine tune it, or however you want to describe the process is fine, and it could become an interesting part of the published review.

My only point is that the reader should know what the variables are under which the component is reviewed, and whether or not more than one has been changes. I think it changes the outcome of the comparison. If footers, power cords, and/or speaker position are changed without describing the full sonic results, the reviewer is doing a disservice to his readers, and the review, for me, has less value.
Usually list of associated ancillaries gives me good insight to what the reviewer is or isn't hearing, setup and speaker position is one of those things that everyone believes they got it right until they know that they didn't! Without a baseline being told that speakers were repositioned during a review doesn't help with anything.

david
 
Last edited:
  • Like
Reactions: K3RMIT and morricab

andromedaaudio

VIP/Donor
Jan 23, 2011
6,278
1,165
580
Amsterdam holland
In my view marty does it the correct way .
Its mostly audiophiles who buy stuff they just want/ need to have based on pictures / dealer or show set ups / reviews or what ever .
Then when they place it in their own home and connect everything , suddenly something aint right .
Then the whole proces of moving the speakers around , fiddling with cables and trying other amps starts .
Marty decided on the speakers and what then rests is just swapping amps untill you re satisfied, no need to bring in more variables.

Roy gregory happen to have the XVX s " the best speaker " for review .
What you then need is the " best amp " to get the " best sound "
To reach somewhat of a consensus in the high end
Unfortunately it doesnt work that way , its all far to personal regarding assembling a satisfying system
.
 
Last edited:

andromedaaudio

VIP/Donor
Jan 23, 2011
6,278
1,165
580
Amsterdam holland
What i do wanna add is that in a way its a pitty that the new bigger wilson speakers are hard to drive .
For example M fremer and Roy gregory , marc mickaelson are basically stuck with a 1.6 ohm minimum speaker , which limits amp ( reviewing possibility ) choice considerably .

So then another variable you could add to the mix and that is :

Does a XVX / CH system sound better then a X2 / LAMM one???
I dont know you tell me
 
  • Like
Reactions: sbnx

matakana

Well-Known Member
Aug 26, 2020
283
132
50
59
UK
This topic came up in Marty’s amplifier thread. Member cableman questioned Roy Gregory’s review methods. Mr. Gregory stated that he will change speaker positioning within the room to better optimize the sound of various amplifiers being reviewed. My question is: does changing more than one variable at a time change your opinion about the value of the review of the component being compared?

What would you think of a turntable review if the reviewer changed the tonearm and cartridge to optimize the sound of the turntable in a given system context?

This is my response to Ron Resnick in the thread which I deleted because it is off-topic to Marty’s thread.



Ron, the reader is free to believe whatever he wants from a review. Personally, I don’t want to see the reviewer change more than one variable at a time before describing the effect on the sound. The exception I can think of is replacing an entire chain of electronics from one brand to another brand.

I did not know that this was a consistent process for Roy Gregory. I appreciate that cableman pointing it out. Knowing this about Mr. Gregory‘s amplifier reviews has changed my opinion about the validity of the reviews.

Marty did not do that when he made the comparison. I did not do it when I compared two different electronic chains in my system. Knowing the review process is helpful because it helps us place value on the results of the review.
If everyones loudspeakers weighed in at say 20 kg each, then likely we would not be having this discussion since when one changes out an amp or whatever, it would be part and parcel in trying a different placement for ones loudspeaker in finding the right bounce should the soundscape be off in a wee way! IN OTHER WORDS IT WOULD BE STANDARD PRACTICE.
 

PeterA

Well-Known Member
Dec 7, 2011
9,496
6,376
1,335
North Shore of Boston
If everyones loudspeakers weighed in at say 20 kg each, then likely we would not be having this discussion since when one changes out an amp or whatever, it would be part and parcel in trying a different placement for ones loudspeaker in finding the right bounce should the soundscape be off in a wee way! IN OTHER WORDS IT WOULD BE STANDARD PRACTICE.

Do you think reviewers should have five or six different power cords on hand for when they review different amplifiers so as to determine which has the best “synergy”?

Are you suggesting that Marty made the determination on his CH M 10 amplifiers prematurely because he did not do exhaustive speaker repositioning before he decided he did not like his amplifiers?
 
  • Like
Reactions: Cableman

Al M.

VIP/Donor
Sep 10, 2013
7,135
2,740
603
Greater Boston
If everyones loudspeakers weighed in at say 20 kg each, then likely we would not be having this discussion since when one changes out an amp or whatever, it would be part and parcel in trying a different placement for ones loudspeaker in finding the right bounce should the soundscape be off in a wee way! IN OTHER WORDS IT WOULD BE STANDARD PRACTICE.

You hit the nail on the head.
 
  • Like
Reactions: matakana

matakana

Well-Known Member
Aug 26, 2020
283
132
50
59
UK
Do you think reviewers should have five or six different power cords on hand for when they review different amplifiers so as to determine which has the best “synergy”?

Are you suggesting that Marty made the determination on his CH M 10 amplifiers prematurely because he did not do exhaustive speaker repositioning before he decided he did not like his amplifiers?
I have not mentioned any powerchords, re positioning 20kg loudspeakers is free and a no brainer if the sound is off , and yes Marty should have tried repositioning , of course with his dealer that sold him the wilson, they cost plenty of brass ! He gave up too quickly, we still dont know how many hrs were on the M10!!!
 
  • Like
Reactions: Scott Naylor

microstrip

VIP/Donor
May 30, 2010
18,807
3,127
810
Portugal
Of course you can. It's not the way I would approach an experiment from a scientific perspective, but nobody says you can't do it. Knock yourself out! Do whatever you think will get you to the promised land. As far as believing that an amplifier will change the acoustic properties of room resonance is concerned (and hence, speaker placement), I'll just have to respectfully disagree. Tell that to acousticians and watch them laugh at you.

Acousticians will probably assume that all the amplifiers have high damping. However people in WBF are addressing amplifiers with very diverse damping - sometimes lower than 5! This difference can be enough to change the room resonance with a particular speaker.

Surely scientific perspectives are needed to carry scientific experiments, but IMHO listening evaluations seldom have such characteristics. The first and immediate reason is the lack of time to carry all the needed permutations and systematic listening. The second is that we do not typically listen under unbiased conditions, why pretending our listening is scientific?

All we can aim is to a methodology that brings us a better probability of success.
 

About us

  • What’s Best Forum is THE forum for high end audio, product reviews, advice and sharing experiences on the best of everything else. This is THE place where audiophiles and audio companies discuss vintage, contemporary and new audio products, music servers, music streamers, computer audio, digital-to-analog converters, turntables, phono stages, cartridges, reel-to-reel tape machines, speakers, headphones and tube and solid-state amplification. Founded in 2010 What’s Best Forum invites intelligent and courteous people of all interests and backgrounds to describe and discuss the best of everything. From beginners to life-long hobbyists to industry professionals, we enjoy learning about new things and meeting new people, and participating in spirited debates.

Quick Navigation

User Menu

Steve Williams
Site Founder | Site Owner | Administrator
Ron Resnick
Site Co-Owner | Administrator
Julian (The Fixer)
Website Build | Marketing Managersing