ABX testing

Vincent Kars

WBF Technical Expert: Computer Audio
Jul 1, 2010
860
1
0
Hearing is believing.
Unfortunately the reverse is also true, what you believe you will hear.
An argument commonly used by the debunkers to explain why audiophiles hear differences between e.g. cables.
This is called expectation bias.
As far as I know, nobody has ever said that expectation bias works one way.
As a consequence, what you don’t believe you won’t hear.
So much for the debunkers.

The solution is an obvious one, do your listening test unsighted.
This removes your expectation bias.
As you have no clue about what is playing, you can fully focus on where it is all about, the sound and the sound only.

A very pragmatically but highly relevant question is how to conduct your unsighted test.
Anybody familiar with science knows that the right experimental design (and the right experimental setup) is crucial.
The wrong experimental design will invalidated the experiment by design.

A methodology often recommended is ABX.
You listen to A, you listen to B and then somebody randomly plays A or B.
For you this is the X and you have to identify if X=A or X=B.
A couple of trials are needed.
Each time you have a 50% change to guess right.
But to guess right 2 times in a row has a likelihood of .5x.5=.25, 3 times=.5x.5x.5=.125, etc.

Maybe it is nice to run an ABX test now.
Just try this one: http://www.sieveking-sound.de/abx/

Basically what you have to do is:
- detecting a difference
- labelling this difference correctly
If you do see a difference between X1 and X2 but you already forgot the exact difference between A and B, you do detect a difference but your labelling will be at change level.

Obvious if this depends on short time memory (our auditory memory belong to it for small details) there is the likelihood that you do notice a difference but due to wrong labelling the score will be not significant.

I have the feeling that ABX might yield to many false negatives.
It might be the wrong experimental design for detecting small differences.
 
Last edited:

Phelonious Ponk

New Member
Jun 30, 2010
8,677
23
0
Vincent what makes you think AB/X might lead to many false positives?

Tim
 

terryj

New Member
Jul 4, 2010
512
0
0
bathurst NSW
As a consequence, what you don’t believe you won’t hear.
So much for the debunkers.

Why?

I agree that is THE reason why it may not be accepted if I said 'I did not hear a difference'

All true. It MAY form the basis of any audio belief I have (not hearing it myself I mean) but yes, it would not be proof for the other person.

So what happens is the reverse. It is the person making the claim that is required to 'prove' that claim. After all, it IS him that hears it no?

So it is not the debunker that is under test.

It is the claim that is under test.
 

Phelonious Ponk

New Member
Jun 30, 2010
8,677
23
0
Why?

I agree that is THE reason why it may not be accepted if I said 'I did not hear a difference'

All true. It MAY form the basis of any audio belief I have (not hearing it myself I mean) but yes, it would not be proof for the other person.

So what happens is the reverse. It is the person making the claim that is required to 'prove' that claim. After all, it IS him that hears it no?

So it is not the debunker that is under test.

It is the claim that is under test.

We can hear what we want to hear. We can not hear what we don't want to hear. Expectation bias works both ways. But I see no reason to believe in more of one than the other, beyond the mix of the participants; skeptics are much more likely to conduct casual AB/X tests. In formal, statistical testing, false positives/negatives are addressed by the margin of error. Casual, personal testing? I find other people's test interesting, but I look for no proof of anything in anyone's tests but my own. In my own I find compelling evidence of whether or not it matters to me.

Tim
 

Vincent Kars

WBF Technical Expert: Computer Audio
Jul 1, 2010
860
1
0
I did the test
What struck me was that when presented with A and B it is easy to note the differences.
But even at the first trial I had great trouble to identify X correctly.
Maybe it is specific for me (not very good in colors) but even this small 5 seconds delay is obvious too much for me.

Eager to hear your experiences
 

terryj

New Member
Jul 4, 2010
512
0
0
bathurst NSW
Ok, first off, been watching the semi final of the rugby, had a few beers, but what the heck eh?

Si I did click on the link.

A colour test came up. Hmm, interesting.

What has that to do with hearing?

Alright, click a few times, expected some new screen to come up (it didn't, it was not till later 'after nothing happened' I noticed that the count had increased).

So I decided to click on 'I did see a difference' and found that it was now another visua) test, yet had to wait five seconds ( I think it was)

Al;right, I also notice it was an 'audio' site.

So, even half drunk let me make some assumptions. (what do they sell? Cables? Isolation cones? The suspense is killing me).

So, am I to assume, the whole point is to illustrate WHY DBT's are poor in revealing actual sonic differences? (again, what do they sell?)

No wonder *we* always request quick switching eh. WHO are the ones that tell us 'quick switching is NOT how we listen'???

Crikey, talk about setting up a strawman.

It's like one of thse card tricks..the guy KNOWS where the card is, and depending on which the other person picks he knows which card to subtract. Of course, the end result is what the trickster fully intends it to be.

And some call this a valid illustration of the downsides of abx tests???????

THAT is the wonderment!
 

Johnny Vinyl

Member Sponsor & WBF Founding Member
May 16, 2010
8,570
51
38
Calgary, AB
11/20.......
 

Phelonious Ponk

New Member
Jun 30, 2010
8,677
23
0
These tests have no relationship to testing of audio at all. In the AB test, there is no switching. This is impossible in audio. In the AB/X test the 5 second delay is way too long for very small differences in either color or audio.

Given immediate switching, you might be able to discern very small differences. Given a 5 second gap, the smallest differences would probably be very hard to detect. Given a couple of weeks with a beloved new component and we can talk ourselves into almost anything. But this "test" is completely irrelevant to audio and obviously created by someone wanting to demonstrate that AB/X testing is a "stressing and frustrating experience."

Tim
 

Johnny Vinyl

Member Sponsor & WBF Founding Member
May 16, 2010
8,570
51
38
Calgary, AB
May we hear an "Amen" at this point? !!

VERY well said. I concur completely !!!

I agree as well!

Has anyone ever bought a component/accessory based on the results of these tests?
 

Phelonious Ponk

New Member
Jun 30, 2010
8,677
23
0
For choosing between two components, fast switching is all I need as I do know that price does not mean anything when it comes to what my ears hear.

Same here. I always got the components in-house and set up to switch between them blind. No X. I understand that I didn't reach any statistically valid conclusions, but I satisfied myself as to whether or not the difference, if I heard one at all, was preferred and worth the investment. I think good AB/X testing should be an excellent tool for debunking audiophile myths, the problem is that the believers just choose to continue to believe the myths and deny the science.

Tim
 

amirm

Banned
Apr 2, 2010
15,813
37
0
Seattle, WA
Clever test :).

Going to audio, I have pretty direct experience with the impact of delay. There have been countless cases where I could not tell the difference with long switch over of a few seconds (and got very frustrated with the delay) but could spot it with millisecond delays. I usually find the issue on a note that lasts a fraction of a second. Being able to loop on that and switch back and forth hugely increases the detection limit.

When I sat through Harman speaker tests, the delay was 4 to 5 seconds. I would get quite anxious waiting for the next sample. So as much as that is a huge advance in speaker testing, it definitely impacts the ability of how much one can detect even if the differences are big.
 

Ron Party

WBF Founding Member
Apr 30, 2010
2,457
13
0
Oakland, CA
Amir, you're a fortunate man to have visited Sean, Floyd, et al., and to have been afforded those opportunities. I hope to be as fortunate very soon.
 

Orb

New Member
Sep 8, 2010
3,010
2
0
Clever test :).

Going to audio, I have pretty direct experience with the impact of delay. There have been countless cases where I could not tell the difference with long switch over of a few seconds (and got very frustrated with the delay) but could spot it with millisecond delays. I usually find the issue on a note that lasts a fraction of a second. Being able to loop on that and switch back and forth hugely increases the detection limit.

When I sat through Harman speaker tests, the delay was 4 to 5 seconds. I would get quite anxious waiting for the next sample. So as much as that is a huge advance in speaker testing, it definitely impacts the ability of how much one can detect even if the differences are big.

You touch on something I am pretty critical of in a lot of the audio testing; that being able to do a satisfactory loop (selected and defined by the listener) on a small segment (from say half a second up to X seconds) of the played music and also switch between them while maintaining and focusing on the loop.

The other part that has me wondering about the AB/X test and studies relates to the delay in switching and how many times the listen is repeated before deciding.
I wonder if there is a relationship between the length of delay and how many times a listener should repeat the listening before deciding, meaning there should be a minimum amount of repeated loop listens (whether a small or whole segment of the test piece) that increases as there is more delay in switching.

I do not think there has been any studies looking at the possibility of selection perception also being affected by the time spent focusing on repeat listen AB/X before deciding what X is (beyond what people say of the 5-10 second rule that appears a lot on the forums).
What has got me curious is that an online UK review site did a test with 4 listeners at a show using blind test protocol that suffered lengthy delay as the listeners left the room while a person physically changed the cables and once that person left they came back in (to stop subtle cue information being picked up).
What is interesting that there was a lengthy delay in being able to switch cables due to this, but one of the listeners wanted to repeat the listen many times before deciding what X was and his result was 100% while the others were average to poor (20% correct).
The downside is that they only managed 6 actual test cycles which is far short of 12, and the test still took 2 days (about 4-6 hours there in total).
Not enough to have any statistical implications, but does get me wondering if there is a relationship between the capacity to be able to focus on listening for a long length of time just for one test cycle (so usually would be 12 test cycles with different music for each to have some meaning) and switching between AB/X many times before making a decision what is X, and critically be able to repeat this for the next 11+ cycles.

Cheers
Orb
 

About us

  • What’s Best Forum is THE forum for high end audio, product reviews, advice and sharing experiences on the best of everything else. This is THE place where audiophiles and audio companies discuss vintage, contemporary and new audio products, music servers, music streamers, computer audio, digital-to-analog converters, turntables, phono stages, cartridges, reel-to-reel tape machines, speakers, headphones and tube and solid-state amplification. Founded in 2010 What’s Best Forum invites intelligent and courteous people of all interests and backgrounds to describe and discuss the best of everything. From beginners to life-long hobbyists to industry professionals, we enjoy learning about new things and meeting new people, and participating in spirited debates.

Quick Navigation

User Menu

Steve Williams
Site Founder | Site Owner | Administrator
Ron Resnick
Site Co-Owner | Administrator
Julian (The Fixer)
Website Build | Marketing Managersing