What is Your Test for Comparing Two Audio Components?

[bonzo75]

What about a mandatory hearing test for reviewers.
That would surely help the industry

Then you would need an audio test for those who test the reviewers.

I think whether reviewers can hear or not, can be made out by carefully analysing reviews and benchmarking some of that experience yourself

 

[Iamrael]

different part of the brain working? don't know really. maybe more that we find ways to be fully engaged to what we 'HEAR' and 'FEEL'. and not to what we see or any baggage we bring to the listening that might cause a bias. such as the stress of blind listening and some sort of outside expectations. pressure to measure up to other's perceptions. or be 'right'.

with all due respect, to begin with, i don't regard blind/double-blind testing as useful for hifi. not to say that in product development it can't work in some form or fashion for a designer under very rigorous conditions. but audiophile listening and the science of testing are separate animals.

our actual method of listening as far as eyes open or closed or any personal preference like that is a separate subject from blind testing. my opinion is that we all find our own methods that seem to work best over time that hold up under long term listening. we can be tempted to just believe first impressions and then proceed. it's most important to revisit conclusions over time for confirmation. especially how the music effects us and how we are immersed. hard to get that with quick hits in blind testing.

i do think over time we learn to listen better, to realize what is important to us in the music and the nuances and tells we pick up on. this past week a WBF member visited me and spent two days listening. he had heard the Grateful Dead/Jerry Garcia 300 times in person, over 30 years, and did lots of taping. we played lots of streaming files he had been present at and he had his own set of things he clued in on. it was his truth. that is very personal. i really could not relate to where he was coming from. my reference for his music was not very relevant. he did have to adjust some of those experiences to not being medicated. he said something about 'Bear' providing the best quality meds.

Thanks for your response Mike!
For the record I never made any claims regarding your position on blind testing.
I have never done any formal blind testing myself that has led to anything meaningful.
I have always thought that blind testing works only for people who listen to their system blind/eyes closed.
I have 9 of overhead lights in my listening area all with Hue bulbs. Combined with the “ON” iOS app I have programmed different scenes to influence my perception when listening. The effect is not even close to being subtle.

 

[Kingrex]

That's why you need time to do it properly to acquire a stable understanding across your varying moods and states. There is the need to live with both products across days to understand their sonics and ergonomics.

When I do a formal product review I will take 2-4 weeks to research the product -- knowledge, book-learning. I try to learn everything I can from the manufacturer, understand how it works, what is special about its features, options etc. During that time I use the product to get it burned-in, only listening to it at intervals to gauge the burn-in progress.

Then I will take somewhere between 4-6 weeks using it in my system at 1 - 4 hrs a day. I take notes during each listening session. I re-listen to recordings to compare my notes from, say, a couple weeks ago. Typically, the compared-to product is something I already have and know well. I will spend at least two weeks doing listening comparisons, again taking notes as I go. Then I write.

Do you ever find yourself having to consciously put yourself in review mode?
I find listening for a job and listening as a consumer are sort of different. When I'm working and listening, my head goes to another place. At times I have to get myself, into the space. That's part of why I stopped grain orienting wire. It took a lot of concentration. It took fous. It took time to set up, get your head in a space, then listen. And I had to do it multiple times over a week to ensure I was getting the same answer every time.

When I'm listening for fun, I perform changes when I get an itch. Like the other night. Vocals were a little forward, so I dug out a different interconnect and plugged it in. I immediately heard the voices recede a little. And a little haze. Then I left it in for the last couple days to see what I feel like over time. Last night I was comfortable, but stull aware there was a little veil.

 

[Kingrex]

I have suggested every reviewer for the major publications have me come over, inspet, clean and repair if necessary their electrical. It was dramatic the change at Fremers when I did his house. Some would say the baseline just changed. But there are other characteristics that come out when the noise from old, dirty, poorly laid out electrical is fixed. I have looked at other reviewers home electrical and some is very lacking and just wrong. Others is decent.

 

[Mike Lavigne]

Thanks for your response Mike!
For the record I never made any claims regarding your position on blind testing.

i had zero impression that you did and my serious reaction is that it's just a hot button issue for me. i did some of it (very casually administered) decades ago and found it somewhat stressed me for whatever reason, and so it was really not conducive for my mind to be free. but maybe in a professional clinical case where there is no right outcome it might be different.

i do like 'accidental' blind testing where we hear something wrong or different in our system, and then find something loose, or wrong, or disconnected. that stuff is true 'proof of concept'.

i have been a judge in our local hifi club speaker building contest for 20 years, every other or third year. the speakers are set up behind a drape and there are three judges and we fill out a paper with various performance criteria. the hard part is there are typically 8 or 10 or up to 15 sets of speakers, it takes time to switch them, and it becomes a real grind to maintain fresh ears and heads to be fair and engaged. it's hard work to listen and concentrate when some of the speakers are a bit of a mess. but some are really good too. so that is blind testing.

I have never done any formal blind testing myself that has led to anything meaningful.
I have always thought that blind testing works only for people who listen to their system blind/eyes closed.

interesting. i never related the two issues before but i do see your point.

I have 9 of overhead lights in my listening area all with Hue bulbs. Combined with the “ON” iOS app I have programmed different scenes to influence my perception when listening. The effect is not even close to being subtle.

i had to google 'hue bulbs'. thanks. that might be neat. i have a Lutron panel and have different light settings for different use cases. but mostly it's either dim or 'reading/surfing' where the speaker end is dim and the people end is back lit.

 

[Kingrex]

it's just a hot button issue for me. i did some of it (very casually administered) decades ago and found it somewhat stressed me for whatever reason, and so it was really not conducive for my mind to be free. but maybe in a professional clinical case where there is no right outcome it might be different.

I believe a major contributor to this stress is that when sighted, a person may think they hear something, and mentally they convince themselves they prefer it. But when unsure the parameters, they worry they may actually find they prefer the other, and that goes against their conscious decisions/bias.

I agree, blind test are much more stressful than casual A/B. And A/B is more stressful than set it, and forget it for a week and see if you find yourself engaged. But its all part of something to do, especially if your a business, to ensure your using all methods available to understand what your hearing. You need all 3.

As a business, I have to find ways to Make myself Blind. I need to remove my personal bias. Personal bias can be a big fat lie. A well executed blind comparison takes personal bias out.

I am not saying you do this Mike. But I do believe many audiophile get caught up with bling and price and mentally convince themselves they like the lesser of 2 products. Where a blind test, done well, would keep them from purchasing something new and supposedly, an upgrade. When really its just sonically a little different . Not better, not worse. But a whole lot more money.

 

[Iamrael]

i had to google 'hue bulbs'. thanks. that might be neat. i have a Lutron panel and have different light settings for different use cases. but mostly it's either dim or 'reading/surfing' where the speaker end is dim and the people end is back lit.

It’s virtually limitless wrt intensity and color palette.
Either subtle or extreme.

 

[Ron Resnick]

i did some [blind testing] (very casually administered) decades ago and found it somewhat stressed me for whatever reason

I agree. I think audiophiles find blind testing to be stressful.

As audiophiles we expect ourselves to be able to hear things and to differentiate between components and sounds. By introducing a type of rigor (not necessarily a dispositive type of rigor or a scientifically valid answer) blind testing challenges our biases, our assumptions and our listening skills. That's why it's stressful.

We stress about whether our hearing and our perception is good enough to "get it right."

 

[Mike Lavigne]

It’s virtually limitless wrt intensity and color palette.
Either subtle or extreme.

pretty cool. at some point might like to try it and see how it works for me.

is the switching hard wired into your room? or something ancillary?

 

[Kingrex]

It’s virtually limitless wrt intensity and color palette.
Either subtle or extreme.View attachment 157386View attachment 157387

LED. Look at the shadowing.

 

[Kingrex]

pretty cool. at some point might like to try it and see how it works for me.

is the switching hard wired into your room? or something ancillary?

Color LED bulbs and remote control of bulbs is widely available. The issue is at times 1 or morr bulbs may respond different to the remote signal. Especially with dimming. Some may dim when you hit the button and others may not. It can be hard to balance them.

Color is less an issue. If they don't all change to a fixed frequency when you hit the button, hit it again and the remainder will adjust.

I would rather find good Edison bulbs that are incandescent and use those.
I also like halogen style MR16. But you still have a transformer.
Incandescent flood, PAR20 are the best, but getting harder to find. I have a case. Always use soft start switch/dimmer with any incandescent and you can get 20 years out of them.

 

[Mike Lavigne]

LED. Look at the shadowing.

can't handle the LED look inside up to this point.

 

[mitch2+]

Then you will always be limited by what you did not know at the beginning of your journey. Because you put up your first system then. All your comparisons/what you might consider progress, was done based on that system.

No argument. The approach works better in the context of a mature system, where you have already achieved most of what you wanted to accomplish. To the OP's question, my mindset probably went directly to digital components/improvements, which is where I have been focused for the past couple of years.

 

[microstrip]

(...) When I do a formal product review I will take 2-4 weeks to research the product -- knowledge, book-learning. I try to learn everything I can from the manufacturer, understand how it works, what is special about its features, options etc. (...)

Unfortunately IMO what reviewers get from the manufacturer is mostly specifications, marketing and some nice drama, that they will reproduce without asking him any tricky questions.

During that time I use the product to get it burned-in, only listening to it at intervals to gauge the burn-in progress.

Some people will tell that we are slowly getting used to its sound during burn-in.

Then I will take somewhere between 4-6 weeks using it in my system at 1 - 4 hrs a day. I take notes during each listening session. I re-listen to recordings to compare my notes from, say, a couple weeks ago. Typically, the compared-to product is something I already have and know well. I will spend at least two weeks doing listening comparisons, again taking notes as I go. Then I write.

Nice to know about your work.

 

[bonzo75]

No argument. The approach works better in the context of a mature system, where you have already achieved most of what you wanted to accomplish.

So how did process to get to a mature system differ? And is it the owner who decides his system has now matured?

my mindset probably went directly to digital components/improvements, which is where I have been focused for the past couple of years.

That isn't mature, that will happen when you get to analog

 

[Lagonda]

Really, I hate gold hi-fi equipment. It's just as bad as chrome trim on modern cars. I know, I'm swimming against the current again.
the only place where gold likes is in my bank safe deposit box

I find the gold a little tacky too, but still love my MBL gear.

 

[microstrip]

(...) with all due respect, to begin with, i don't regard blind/double-blind testing as useful for hifi. not to say that in product development it can't work in some form or fashion for a designer under very rigorous conditions. but audiophile listening and the science of testing are separate animals. (...)

In fact blind listening is not running an A/B test in ridiculous conditions, as typically reported in audio forums. Blind testing is a proper method, that needs resources, knowledge and time. Better avoiding it if we do not master it or do not have the conditions to use it.

our actual method of listening as far as eyes open or closed or any personal preference like that is a separate subject from blind testing. my opinion is that we all find our own methods that seem to work best over time that hold up under long term listening. we can be tempted to just believe first impressions and then proceed.

Our method is essentially living with our owns biases and using listening in guiding us in an individual search. Surely most times audiophiles are strongly influenced when listening by the people joining them in the listening - IMO it the hardest to avoid factor of bias.

it's most important to revisit conclusions over time for confirmation. especially how the music effects us and how we are immersed. hard to get that with quick hits in blind testing.

"Over time" and compares are not compatible for 99% of the consumers - few people can keep three units for a reasonable time in their listening room.

 

[Lagonda]

Then you would need an audio test for those who test the reviewers.

I think whether reviewers can hear or not, can be made out by carefully analysing reviews and benchmarking some of that experience yourself

Like most of the customers, they are getting older and more deaf year by year, only Fremer was honest enough to post a review of his hearing aid !

 

[Mike Lavigne]

Some people will tell that we are slowly getting used to its sound during burn-in.

hopefully you have a handy alternate reference on hand to help confirm this progress. not always possible, but many times is.

sometimes the break in changes are very very subtle and hard to quantify. settling down. calmer. slight degrees of opening up, clearing up. more definition and articulation, a little freer and greater flow. bass deeper and more weighty. but an alternative reference can expose these things.

In fact blind listening is not running an A/B test in ridiculous conditions, as typically reported in audio forums. Blind testing is a proper method, that needs resources, knowledge and time. Better avoiding it if we do not master it or do not have the conditions to use it.

we do the best we can to navigate the system building challenges. blind testing is not a tool i choose.

Our method is essentially living with our owns biases and using listening in guiding us in an individual search. Surely most times audiophiles are strongly influenced when listening by the people joining them in the listening - IMO it the hardest to avoid factor of bias.

agree. my most frequent listening buddy has quite different listening biases to me. so it's easy to not comingle our conclusions. we might hear the same things, but value them quite separately.

"Over time" and compares are not compatible for 99% of the consumers - few people can keep three units for a reasonable time in their listening room.

this thread relates primarily to source or amp compares, and in those cases i agree not always realistic. but many compares involve gear we might add or subtract, treatments and such, that are easy to confirm later. so it's an approach that can be sensible to confirm.

 

[Lagonda]

So how did process to get to a mature system differ? And is it the owner who decides his system has now matured?



That isn't mature, that will happen when you get to analog

A fully mature system can put me to sleep in under 15 minutes when playing classical music. Powerful stuff !