Perceptually erasing differences

amirm

Banned
Apr 2, 2010
15,813
37
0
Seattle, WA
A lot has been said about how placebo effect helps to manufacture differences in our heads based on personal or experiment bias. In contrast, I have not heard much about the reverse effect: the brain erasing differences that may be there.

Let me expand with a few data points:

To get good sleep when I am travelling, I have learned to bring ear plugs with me. That way, the guy who wakes up at 4:00am to catch an early flight and slams his door next to me, doesn't wake me up. When I use my ear plugs, I start to hear my own pulse. At first it is a distraction. But I have learned to focus on it and try to reduce its rate by relaxing. And by doing so, I quickly fall sleep. So clearly, we have control over our body in ways we don't always appreciate.

By the same token, I have been doing a ton of tests both triggered by analysis work we have been doing in the De-Mag thread and at work, trying to optimize and compare audio components. Since many times these tests are sighted, I try to see if I can nullify the effect of placebo by trying to see if I can nullify the difference I hear just like the heart rate exercise.

This is a skill I learned long time ago which I thought I heard differences in files that were identical. The technique takes a bit of work to develop as was the case with lowering my heartbeat. At first your brain tells you that it can't happen but if you believe it, and let it happen, it indeed works.

In every comparison, I am able to reduce the difference I am hearing by forcing myself to think that way. Put another way, if the observation iis at 100, then I am able to push myself -50 points to the low end. And as we know from placebo effect, I can also imagine it to be bigger by probably the same amount.

So the question then becomes, if the brain is just as good at erasing differences as it is to imagine them, would that not distort blind tests? When I hear tiny differences in blind tests, I am always careful to think whether I am imagining it, knowing well that I may be listing to the hidden reference (both samples being the same). What if I am erasing a difference that is really there, by pushing myself to erase the perceived difference?

If true, then our perception has +- variation of X. It then follows that any difference less than X cannot be reliably determined in such a setting where a person is being challenged as I have been. I suspect even average listeners, when told that they may be arriving at wrong results in testing, will at times attempt to moderate the differential they are hearing and hence, fall in the same bucket as me but perhaps with less variation than I can force myself into.

I realize we are not going to prove anything in this discussion. But want to probe deeper into experiences others have and whether this kind of thinking holds water. Asa fun exercise, I suggest you also go and perform the above test. Take a tweak that you have done recently and see if you can imagine it making less difference than you thought and see if you can erase some or all of the advantages you thought it had.
 

mep

Member Sponsor & WBF Founding Member
Apr 20, 2010
9,481
17
0
Amir-Can we imagine our way into having a $500K stereo system? That would be cool. Seriously, can you define what you mean by a "tweak?" Personally, following the defnition of what I thnk a tweak is, I don't find myself to be much of a tweaker. Once I set up my system, other than performing routine maintenance (checking/setting the bias on output tubes, cleaning the spindle on my air bearing arm, making sure the table is still in level, cleaning my tape heads, etc.), I pretty much just sit back and listen to music. I don't have cable elevators, hockey pucks, or a Furtech Demag unit.

Now I recently bought a sub from the same company as my main speakers and integrated that into my system. Does that count as a tweak?
 

amirm

Banned
Apr 2, 2010
15,813
37
0
Seattle, WA
By tweak I meant something that can be added and deleted to a system and one that makes a small difference. So cables would be one. Isolations means would be another.

No, sub would not count as that makes a significant difference that would fall outside of this theory and exercise.

As to thinking you have a $500K system, that is not in the cards :). But you would be surprised how easy it is, for me at least, to read "more air" into instruments and such on a good system. When the effect/performance is not there at all, it is harder to imagine it. But once some of it is there, exaggerating is very easy for me. So maybe I can make a $50K system be like a $70K one :p.
 

Kal Rubinson

Well-Known Member
May 4, 2010
2,360
697
1,700
NYC
www.stereophile.com
Your perceptions are never a true indicator of reality but a transform filtered through brain mechanisms involving mood, conscious bias, subconscious pattern matching and a myriad of ill-defined issues that result from experience. IMHO, what you are describing is the limited effect of imposing some conscious bias on the situation.
 

mep

Member Sponsor & WBF Founding Member
Apr 20, 2010
9,481
17
0
Amir-If I was King for the day, I would define a tweak as something that you can remove from your system or room and the stereo system will still function normally. For example, I need cables. I can't remove them and have my system function correctly. If you ask me to remove my cables and reinsert the old ones, then that is not much different then saying maybe the new preamp you just bought isn't really much better than your old one so pull out the new preamp and reinsert the old one. Bottom line, I wouldn't consider cables tweaks. You can remove hockey pucks, cable elevators/insulators, mpingo discs and a whole bunch of other things like that and your system will still operate.
 

amirm

Banned
Apr 2, 2010
15,813
37
0
Seattle, WA
Well for my definition, assuming you still have your old cables, it would do. Put the old one in. Listen to them. Then put the new one and try to imagine they make no difference at all.
 

mep

Member Sponsor & WBF Founding Member
Apr 20, 2010
9,481
17
0
Drats. I knew I couldn't be King for the day. I still have my old cables, but I'm too lazy to remove my new speaker cables, find the jumper plates to reconnect the bi-wire terminals together, and hook up the old cables, and then use my imagination to see if Ethan-like, I can null out the differences in my mind so that they both sound identical. And honestly, I don't think the differences between my old speaker cables and new speaker cables are of the night and day variety in the first place. I do like the fact that these are bi-wire cables and I was able to remove the jumper plates from the speakers and drive them as the designer intended.

Hopefully someone else will undo something they did and reinsert the old thingy and imagine that they made no improvement to their system and wish they had their money back that they wasted.
 

garylkoh

WBF Technical Expert (Speakers & Audio Equipment)
Sep 6, 2010
5,599
225
1,190
Seattle, WA
www.genesisloudspeakers.com
I think that we don't even have to consciously reduce the difference if it's slight and subtle. I might have mentioned it before here or somewhere else, but some time ago I was doing a large number of ABX comparisons using two files. I did trails of 20 samples each. The first trial I did - I got well over 80% correct. But the more I did with the same test, the less and less difference I heard until I reached the point where I could not hear a difference, and was essentially guessing.

Was the difference I thought I could hear in the first trial expectation bias and the 80% score luck? I don't think so because the score went down gradually over a number of trials. If it was luck, it should have gone up and down randomly.
 

amirm

Banned
Apr 2, 2010
15,813
37
0
Seattle, WA
Drats. I knew I couldn't be King for the day. I still have my old cables, but I'm too lazy to remove my new speaker cables, find the jumper plates to reconnect the bi-wire terminals together, and hook up the old cables, and then use my imagination to see if Ethan-like, I can null out the differences in my mind so that they both sound identical. And honestly, I don't think the differences between my old speaker cables and new speaker cables are of the night and day variety in the first place. I do like the fact that these are bi-wire cables and I was able to remove the jumper plates from the speakers and drive them as the designer intended.

Hopefully someone else will undo something they did and reinsert the old thingy and imagine that they made no improvement to their system and wish they had their money back that they wasted.
You do know that this thread is in defense of subjective evaluation of audio as opposed to objective in the way it proposes failure to hear meaningful differences. Yes? If so, I am not clear about your comments above.
 

amirm

Banned
Apr 2, 2010
15,813
37
0
Seattle, WA
Was the difference I thought I could hear in the first trial expectation bias and the 80% score luck? I don't think so because the score went down gradually over a number of trials. If it was luck, it should have gone up and down randomly.
This is what I am trying to quantify. We know from firm evidence that if truth is at 100%, we can imagine it through bias to be 150% (placebo effect). What I am trying to investigate is whether it also works the other way in that it tells us the truth is at 80%. If true, then DBTs under-report and under-value positive differences.

Of course I made up the above numbers. But wonder, what they could be. They problem we have is what you faced: how do we know the truth? If the truth was that there was a difference between files, then you fell victim to the theory at hand. You talked yourself out of a difference that was there and heard in a different mindset.
 

mep

Member Sponsor & WBF Founding Member
Apr 20, 2010
9,481
17
0
Well for my definition, assuming you still have your old cables, it would do. Put the old one in. Listen to them. Then put the new one and try to imagine they make no difference at all.

Amir-Your above quote is what you said to do. So what if you can imagine that the cables made no difference in the sound and your old cables sounded just the same as the new ones? Maybe this is one test that you don't have to imagine very hard in order to come to that conclusion. What does that prove? It doesn't prove that there is no difference between the cables, it just "proves" that you have a vivid imaganation unless of course there really is no difference.
 

amirm

Banned
Apr 2, 2010
15,813
37
0
Seattle, WA
Amir-Your above quote is what you said to do. So what if you can imagine that the cables made no difference in the sound and your old cables sounded just the same as the new ones? Maybe this is one test that you don't have to imagine very hard in order to come to that conclusion. What does that prove? It doesn't prove that there is no difference between the cables, it just "proves" that you have a vivid imaganation unless of course there really is no difference.
You are still reading this argument in reverse. I am saying let's accept that the difference exists between cables. Then see if you can make it go away in your mind. What this proves is that you can control your mind at least to that degree to hear and more importantly in this context, not to hear!

The "vivid imagination" is your friend here, not enemy. If imagination can wipe out real differences, then it also clouds the results of blind tests where such motivation does exist at times.

Yes, there is a fly in the ointment in that we don't know for sure if the cable does make a difference. But that is for another discussion and argument. I am simply after the power of the mind to negatively influence fidelity, as opposed to infinitely repeated, positive.
 

Bruce B

WBF Founding Member, Pro Audio Production Member
Apr 25, 2010
7,006
512
1,740
Snohomish, WA
www.pugetsoundstudios.com
Some of the differences I have picked up on sound in my room is when I have NOT been paying attention. I, for so long, have be used to sounds and what my system could do. If I'm reading or cruising the internet and listening to a tape or file transfer in the background, I can pick up on artifacts and what's not supposed to be there almost better if I'm distracted or doing something else.
You get so used to your surroundings and what they sound like that any thing out of the norm could and should be easily discernable. Just like in the medical field. You see so many "normals" during the day that when something is "abnormal" you instantly pick up on it.
 
Hi amirm
What you want / expect strongly effects an evaluation of something, that is the reason for blind testing.
As you propose, a person that was inclined NOT to hear any differences would be less likely to hear any, I suppose the “truly inspired” could pick up a dog end and smoke it as if a fine cigar .

The reason why blind testing shouldn’t be ignored is because of the significant bias present when one expends money or effort on something.
For that reason, it would bet that this bias is much more common than a negative bias.

Sadly there is an jewelry or trinket element in hifi which aims at the people who would like to improve , they aim at the small number of things the end user could change that wouldn’t harm the actual operation. That is why we have costly exotic knobs, little rocks and a myriad of add on things that don’t actually alter anything. I know that probably sounds a bit harsh so I will show a couple examples where this same approach is used in other areas.

Avoiding specific things in audio, a humorous example was the Kenoki foot pad, touted as drawing out all the bad stuff through your feet and the proof was there in the stain.
It turned out the inside of the pad contained ash which soaked through the pad when it got moist, it did nothing except make you feel like it did something and of course the object, to get your loose cash..

The next step up are things which can have an effect via a completely different mechanism than what is used to market it. A cool example of this kind of pseudo effect is the gas line magnetizer, you clip a neo magnet on your gas line and most people really DO get better gas mileage.

The scientists / engineers say the magnet has no effect at all on the gasoline, its effect can be measured on a dynamo, zilch Nada no improvement at all..
Why do the results contradict?
The act of buying one and installing it that same mental investment and decision, here you expect to get better gas mileage and so unconsciously, you drive more conservatively and you really do get better mileage.

Into the actual, If one were up chasing small actual differences like cables or electronics, it might be better to use an ABX test instead. Here you know the first one is cable A, the next one you know is cable B and with X, it is your task to determine if that one sounds more like A or B (and is either A or B).
Anyway, some thoughts,
Best,
Tom Danley
 

amirm

Banned
Apr 2, 2010
15,813
37
0
Seattle, WA
Thanks for the comments Tom. There is no question that power of suggestion exists. What I am trying to investigate is whether it goes both ways. DBT indeed eliminates bias. But does it go overboard and erase some true differences also? I think it does.

BTW, I am a strong proponent of DBT as a super useful tool. I have run many of them and participated in just as many. I have been caught with my pants down so to say having been proven wrong :). So nothing here is to attempt to disprove the value of DBT but rather, try to understand its limitations in extracting the truth in the experiment.

DBTs have essentially eliminated any possibility of any high-end product having real benefits. I wonder if it provides too pessimistic view of the data given my observation above.
 

Vincent Kars

WBF Technical Expert: Computer Audio
Jul 1, 2010
860
1
0
I have not studied psychology, only read a couple of books on perception.
One thing I do think is obvious, expectation bias works.
What you believe, you hear.
I never encountered any evidence that it would work one way.
I do think it highly probably that the audiophile will hear differences which are not there and the debunker will not hear differences which are there.
No reason to assume that the coloration inherent to perception works one way only.

DBT
This is even more tricky.
All I know is that methodology is very complex.
Talking meta-analyses (the analyses of analyses) lots of research is discarded from the meta analyzes simply because of methodological flaws.
Even if your experimental design is correct, you can make a lot of errors in the experimental setup too.
Methodology is complex, it is a specialist job.

There are a couple of methodologies known to me by name.

ABX: a bit complicated, if you don't hear a difference it is easy, plain guess work.
If you do hear a difference you also have to attribute the difference correctly to A or B.
If you do hear the difference but attribute to A or B in a non-systematic way, the score will be not significant. I do think there is a risk that ABX under estimates the difference due to non systematic attribution errors

DBT
Often mentioned in audiophile discussions but the Double is basically the one conducting the experiment and you wish to avoid he is giving clues to the one tested.
Swapping power cords "blind" might be a hard and tricky job!

MUSHRA
MUlti Stimulus test with Hidden Reference and Anchors is recommended for audio quality assessment by EBU/ITU-R.
Recommended to test small differences between Codecs.

There are probably more experimental designs

What all these methodologies have in common is testing the null hypotheses.
If the test yields a significant difference, you have proven that in this specific experiment and only this specific experiment the null hypotheses was overthrown.

If the test yields no significant difference, you didn't proof that there is no difference.
You only proofed that in this specific experiment no differences where found.
Absence of evidence is not evidence of absence.

Memory span
Once a guy working for Fraunhofer told me when comparing Codecs they use samples of 1-2 seconds in rapid A/B because auditory memory is not able to compare properly when longer time spans are used!

Regardless of the methodological implications, unsighted testing is where it is about
However it would be nice to know the implications of different experimental designs and their aptness for assessing sound quality.
In other words where is Sean Olive?
 

Phelonious Ponk

New Member
Jun 30, 2010
8,677
23
0
Can you talk yourself into not hearing what you don't want to hear? Of course. There is no reason to believe expectation bias only works one way. Will it happen consistently, over a sufficient number of trials, using varying listening material and subjects who don't know the objectives of the tests or care about the outcome? Very different questions.

Actually, unless your subjects somehow know, as in your example above, that they are not supposed to hear a difference, they will tend to hear one whether one exists or not. Why? Because the rules of the game, even in its blindest, most controlled form are play A, B and X and identify which is which. The subject is pre-conditioned to look for a difference; exactly the opposite of what you're proposing. It only works because of the control.

If this is in defense of subjective evaluation, there is no need to defend it. Listen subjectively. Tell us which amp has more air, which has better bass control, which expands the vertical soundstage beyond the boundaries of your floor and ceiling. But listen blind. if you do it knowing that A is Mark Levinson mono blocks and preamp and B is a Pioneer receiver you don't have a "test" at all. And we all know why.

Tim
 

RUR

WBF Founding Member
Apr 20, 2010
647
3
0
SoCal
Can you talk yourself into not hearing what you don't want to hear? Of course. There is no reason to believe expectation bias only works one way. Will it happen consistently, over a sufficient number of trials, using varying listening material and subjects who don't know the objectives of the tests or care about the outcome? Very different questions.
"Subjects who don't know the objectives..." aren't those being described, Tim.

the debunker will not hear differences which are there.
a person that was inclined NOT to hear any differences would be less likely to hear any

Assuming adequate test conditions (a non-trivial exercise), there is a sure-fire way to skew DBT results and that is to, consciously or sub-consciously, not even try.
 

amirm

Banned
Apr 2, 2010
15,813
37
0
Seattle, WA
Can you talk yourself into not hearing what you don't want to hear? Of course. There is no reason to believe expectation bias only works one way. Will it happen consistently, over a sufficient number of trials, using varying listening material and subjects who don't know the objectives of the tests or care about the outcome? Very different questions.
I don't think that is sufficient condition to limit the impact of negative bias. If you are a tester for anything of the sort, and you are being your opinion of which is which, you are constantly in this mode of trying to be "right." You are played two clips: one sounds a hair different. Who wouldn't think, "hmmm, I wonder if there is really no difference here and I am imagining it." I know, those are the very thoughts I had in DBT tests I have been part of :).

Note that it doesn't take a lot of votes like the above to invalidate the person's data. After all, even if you are half right it means nothing. To get high confidence, scores of 75%+ is targeted. All you need is a few tests where the difference is small and the above factor in play and your total score gets invalidated as "chance."

Contrast this with where blind testing is done elsewhere: medicine. Here, we don't rely on patient's psychology to determine if the drug has worked (putting aside mental illnesses). We rely on independent tests of the patient to see if the drug has had a positive impact. Such is not the case in audio testing. And as such, we are in an uncharted territory a bit.

There are things that help some here to the extent they magnify detectability:

1. Having tests are super revealing. You have to think through the mechanism for change and then focus on magnifying it. I once worked for a large computer company in 1980s where we had just developed a $1M mini-computer. They had a hardware test group and the guy took every benchmark and workload he could at it and after running for two weeks, the system would crash with a random failure. Of course, they blamed me saying it was a software issue. I spent a good month trying to find the cause and I just couldn't. I would change something and two weeks later, a random failure would occur. I started to think what aspect of the system I could exaggerate to shorten this time. Turned out increasing the amount of data the system output to terminal screens sharply reduced the crash time. I kept working this angle until I got it to crash every couple of minutes! That quickly led to discovering a hardware failure which would corrupt one location in memory if an interrupt (caused by outputting the text to terminals) arrived during a few microseconds of one instruction. The problem was fixed and I got a big bonus check :).

Such is the issue when I see many audio tests. Just because a title is audiophile it does not mean it is revealing for testing say DACs, amps, speakers, etc. You don't test the cornering of a car by going straight in a beautiful highway or test high-speed stability by going 30 miles an hour around sharp bends.

I used to by 10 to 20 CDs a week when I tested audio codecs for about 2 years. Out of the 5000+ songs, I had maybe 10 that were super revealing and that was it. The rest were far less sensitive to algorithm changes.

2. Trained listeners. To the extent one is able to hear differences easier, then the challenge becomes easier.

3. As Vincent mentioned regarding the FHG test engineer, getting fast switching to work is critical. You can find differences that are impossible otherwise. When I worked at Microsoft, we would have special players that would let us quickly switch from one stream to the other, loop as he suggested, etc. Often the revealing portion of a sound is just 1 to 2 seconds. It might be a single guitar pick, or start of a piano note. I was once asked by Microsoft Research to find their audio watermark they had inserted in a full song. I got the before and after files which were 24-bit, 96 Khz. I had a reputation of being a trained listener at Microsoft so not being able to find that was going to literally shameful :). So I played every 2-5 seconds of each clip and marched forward until I thought I heard something. I then narrowed to a second or so that sounded different. I reported the precise spot to the researcher and his jaw fell down :). I could have never done that without the right tools.

4. Having the best instrument for the job. When testing sources, why not have the best DAC and headphone system to test them? The more revealing the downstream equipment, the easier the job.

With all of these optimizations, I have managed to hear differences blindly that the common wisdom says does not exist. But I am not able to always deploy all of these tools. Such is the case for example in the DeMag thread. I don't know whether the tracks being used are revealing of the effect of magnetism on LPs. So in that sense, we are violating rule #1. Random search for difference in an infinite number of combination is not a good methodology :).

Anyway, rambling on. :) It is good that everyone intuitively accepts that the bias effect works both ways: preference for what we want to sound better and also, to be "right" in such tests.

As to Mark Levinson amp being better than a Pioneer, I just tested our Mark Levinson amp against lessor ones. There is one clear difference: the ML stays cleaner at higher volumes than amps costing less. Clearly it has more headroom given the much larger power supply, and output stage. So putting aside subjective differences at lower volume, a V-8 engine with double the horsepower and torque of a 4 cylinder engine, will make you go faster :).
 

Ethan Winer

Banned
Jul 8, 2010
1,231
3
0
75
New Milford, CT
When I hear tiny differences in blind tests, I am always careful to think whether I am imagining it, knowing well that I may be listing to the hidden reference (both samples being the same). What if I am erasing a difference that is really there, by pushing myself to erase the perceived difference?

As you know, my approach is practical and concerns the "big picture" mostly. Do not take this to be the same as settling for good enough, though I guess it's related. With that said, if a difference is so small that you're not sure there even is a difference, then who cares? A difference that small will not make a great recording sound bad, nor will it reduce the enjoyment of the music which, as is pointed out by both sides of these debates, is what really matters. And from my perspective as a consumerist, is surely not worth spending thousands of dollars extra for.

--Ethan
 

About us

  • What’s Best Forum is THE forum for high end audio, product reviews, advice and sharing experiences on the best of everything else. This is THE place where audiophiles and audio companies discuss vintage, contemporary and new audio products, music servers, music streamers, computer audio, digital-to-analog converters, turntables, phono stages, cartridges, reel-to-reel tape machines, speakers, headphones and tube and solid-state amplification. Founded in 2010 What’s Best Forum invites intelligent and courteous people of all interests and backgrounds to describe and discuss the best of everything. From beginners to life-long hobbyists to industry professionals, we enjoy learning about new things and meeting new people, and participating in spirited debates.

Quick Navigation

User Menu

Steve Williams
Site Founder | Site Owner | Administrator
Ron Resnick
Site Co-Owner | Administrator
Julian (The Fixer)
Website Build | Marketing Managersing