DIscussion of ABX results of Winer's Loopback files

[jkeny]

I brought this over from Amir's high-res ABX results thread as I didn't want to confound matters on that thread

I found this result interesting as the normal approach to ABX testing is a certain training/testing period where a search is done for a section in the two files that sound different to you & this is tested sighted first before doing the ABX test itself. Usually the listener is able to verbalise what (s)he was listening for.

This first result is slightly different in that the listener just did the test intuitively & wasn't sure of his choices but scored a very positive score.
He then went on to do the harder versions of the test

The post is here

Yes! I am a god of 20 pass ADDA shitty soundcard ABXing. Kneel before me and tremble mortals.

foo_abx 1.3.4 report
foobar2000 v1.3.3
2014/07/21 15:22:58

File A: C:\Users\adamd\Dropbox\ew test original\sb20x_original.wav
File B: C:\Users\adamd\Dropbox\ew test original\sb20x_pass20.wav

15:22:58 : Test started.
15:27:35 : 01/01 50.0%
15:29:20 : 02/02 25.0%
15:32:23 : 03/03 12.5%
15:39:20 : 04/04 6.3%
15:42:00 : 04/05 18.8%
15:43:46 : 05/06 10.9%
15:46:56 : 06/07 6.3%
15:47:51 : 07/08 3.5%
15:50:21 : 08/09 2.0%
17:07:51 : 09/10 1.1%
17:08:08 : Test finished.

----------
Total: 9/10 (1.1%)

Ok here is a weird thing. I actually felt very unconfident in my choices and yet they almost all turned out to be right. btw I set out in advance that I was just going to to 10 goes at it and no more. It might sound odd , but I think it is very important to decide in advance what the test is going to be. Otherwise one can fall into a rather subtle form of bias.

Armed with this success I am probably going to have a crack at the 5 pass test. Anyway I did this one on my work computer with bog standard soundcard and logitech speakers. I think I had better have a go at the 5 pass test with something a little more upmarket

 

[jkeny]

And later he posts

In the end I just listened to the second half and concentrated on the highest sort of sparkly treble. The weird thing though is that I did not really feel confident. Maybe you should ahve a go and see whether you intuitive results might be correct.
btw somewhere on the avs thread there is a reference to AK's original files having a 0.4db loudness error. Don't know whether that applies to the ones I listened to..

But then he posts later

it wasn;t really intuitive, I listened to the sodding files about 30 times each pick (A vs B then x versus A and x versus B etc etc). It was the ultimate choice which was intuitive. I never want to hear those awful files again

Then we get this guy, who wasn't able to hear any difference between the 20 pass file & original post

Woohoo!

Quote:
foo_abx 1.3.4 report
foobar2000 v1.3.3
2014/07/21 20:06:34

File A: C:\Users\Dad\Music\sb20x_original.wav
File B: C:\Users\Dad\Music\sb20x_pass20.wav

20:06:34 : Test started.
20:08:28 : 01/01 50.0%
20:08:42 : 02/02 25.0%
20:08:52 : 03/03 12.5%
20:09:22 : 04/04 6.3%
20:10:02 : 05/05 3.1%
20:10:20 : 05/06 10.9%
20:10:48 : 06/07 6.3%
20:12:06 : 07/08 3.5%
20:12:30 : 08/09 2.0%
20:13:09 : 09/10 1.1%
20:13:14 : Test finished.

----------
Total: 9/10 (1.1%)
I actually did the test (out of 10) twice, and only scored 7 the first time, but that was at reasonably low volume. Turned it up to closer to normal levels for the more successful run above.

I must admit I do find this interesting, as it's the first time I can remember "spotting a difference" in a blind test scenario - however contrived/extreme.

I may go back to listen again via Sonos to see if that is masking differences.

But, I may well be over-emphasising what I thought was the intuitive nature of this listening?

 

[maxflinn]

An interesting test but I'm not sure what the fact that a few people could differentiate the various files relatively consistently tells us.

It simply tells me that the artificially created distortion which increased incremantely after more loopback passes was audible.

What is the relevance of this?

 

[amirm]

I too passed Ethan's generational loss tests. Here is my post on AVS:

------------

I've experimented with the audibility of round-tripping audio through the analog domain. With really high quality converters (those in a LynxTWO for example) something like 20+ round trips are possible without any reliably audible effects.

People who want to investigate this for themselves can download relevant test files from http://ethanwiner.com/aes/ under the heading "SoundBlaster Generations". An hour spent with those files and FOOBAR2000 + the ABX plug in will dispell any misapprehensions. This is especially true given that the audio interface that Ethan used performed at a far lower level then a benchmark pro audio interface such as the LynxTWO.

OK guys. You must be playing with me. You really can't tell the difference here? Here are my quick results. I skipped down to 5th generation (five times going from analog to digital and back):
=========
foo_abx 1.3.4 report
foobar2000 v1.3.2
2014/07/18 06:34:21

File A: C:\Users\Amir\Music\Ethan Soundblaster\sb20x_original.wav
File B: C:\Users\Amir\Music\Ethan Soundblaster\sb20x_pass5.wav

06:34:21 : Test started.
06:35:00 : 01/01 50.0%
06:35:10 : 01/02 75.0%
06:35:21 : 01/03 87.5%
06:35:46 : 02/04 68.8%
06:35:58 : 03/05 50.0%
06:36:19 : 03/06 65.6% <----- Difference found
06:36:28 : 04/07 50.0%
06:36:40 : 05/08 36.3%
06:36:51 : 06/09 25.4%
06:37:02 : 07/10 17.2%
06:37:11 : 08/11 11.3%
06:37:25 : 09/12 7.3%
06:37:36 : 10/13 4.6%
06:37:47 : 11/14 2.9%
06:37:58 : 12/15 1.8%
06:38:10 : 13/16 1.1%
06:38:24 : 14/17 0.6%
06:38:34 : 15/18 0.4%
06:38:50 : 16/19 0.2%
06:38:58 : 17/20 0.1%
06:39:12 : 18/21 0.1%
06:39:21 : 19/22 0.0%
06:39:38 : Test finished.

----------
Total: 19/22 (0.0%)

Above I am showing my search for critical section. So when I tested the single generational loss (i.e. "most difficult") I knew what to listen for:

foo_abx 1.3.4 report
foobar2000 v1.3.2
2014/07/18 06:40:07

File A: C:\Users\Amir\Music\Ethan Soundblaster\sb20x_original.wav
File B: C:\Users\Amir\Music\Ethan Soundblaster\sb20x_pass1.wav

06:40:07 : Test started.
06:41:03 : 01/01 50.0%
06:41:16 : 02/02 25.0%
06:41:24 : 03/03 12.5%
06:41:33 : 04/04 6.3%
06:41:53 : 05/05 3.1%
06:42:02 : 06/06 1.6%
06:42:22 : 07/07 0.8%
06:42:34 : 08/08 0.4%
06:42:43 : 09/09 0.2%
06:42:56 : 10/10 0.1%
06:43:08 : 11/11 0.0%
06:43:16 : Test finished.

----------
Total: 11/11 (0.0%)

So I started at 6:34 AM and finished at 6:43 AM for a total of 9 minutes. What I am supposed to do for the rest of the hour Arny?

By the way, I am traveling and the headphone I have is my Shure IEM. So now we have results across three different headphones.

Arny do you honestly not hear the difference???

By the way the techno clip Ethan picked is not ideally suited for this type of testing yet the audible difference is there.

 

[amirm]

An interesting test but I'm not sure what the fact that a few people could differentiate the various files relatively consistently tells us.

It simply tells me that the artificially created distortion which increased incremantely after more loopback passes was audible.

What is the relevance of this?

I think the one thing this and the other tests prove is that there is a wide range of listening ability. This means that unless we deploy critical listeners in such tests, then the results will not apply to them.

 

[amirm]

And later he posts

But then he posts later

Then we get this guy, who wasn't able to hear any difference between the 20 pass file & original post
But, I may well be over-emphasising what I thought was the intuitive nature of this listening?

I think what has been lacking up to now is the encouragement that such tests can be passed. We had biased listeners so much that at the first sign of difficulty, they would give up with a negative outcome. I suspect from here on, people will try harder and differences that do exist will come out.

 

[maxflinn]

Well, one of the guys over on pink fish who did identify the 20 pass file v the original with 100% accuracy (as did I) and the one pass v original with 98.9% accuracy (I think) may have been encouraged to try hard based on reports that others had already done so, 100% successfully. However the same guy recently hosted a properly controlled double blind ABX test of DACs of hugely varying price, and a follow up sighted but level matched test of DACs.

At the beginning of the first test the participants listened to the various DACs sighted first, and several said they heard differences, but blind, none could reliably differentiate them. No differences were found during the follow-up sighted but level matched test either, which I believe many of the same people took part in.

Based on all the reviews, manufacturers claims and user reports of night and day differences between DACs, and the differences observed sighted before blind testing, one might expect that the participants would have been trying just as hard and indeed expecting to spot any differences during the ABX testing of these DACs, and trying even harder in the follow-up test (otherwise, why bother.), and indeed the host proved that by differentiating the one pass file v the original successfully (98.9% success), in the same system (his) that he doesn't have any hearing issues, nor is there any doubt that his system may not be revealing enough.

So whilst there is no doubt that audible differences exist (and are explainable) between Ethan's loopback files, should there be doubt whether differences between the DACs tested previously may have been present but weren't detected? And if so, why?

 

[Ashley James]

This is a more complex subject than you may think. It's true that people's hearing varies, or even that audiophiles may not be very good at recognising right from wrong, just as car enthusiasts may not be the best drivers, but here are other factors than also need to be considered.

In simple terms, the distortion in each part of a system modifies distortion in the others. They all add together, so if you have passive speakers with very high levels of distortion, you may hear differences that you can't with much more revealing and far lower distortion active speakers. Headphones too, don't generally have passive crossovers and are far more clear than typical speakers, so are probably more reliable for this test.

Another factor to consider is that some amps are affected by RF from DACs and others not, so one that is good enough in one system may not be in another with an old amplifier. Those who remember all the fuss when CD first appeared may be aware that the noise was coming from companies making dodgy amplifiers. Philips and Sony didn't have a problem.

I hope this helps

 

[jkeny]

.....

So whilst there is no doubt that audible differences exist (and are explainable) between Ethan's loopback files, should there be doubt whether differences between the DACs tested previously may have been present but weren't detected? And if so, why?

Agreed, you give a fair account of the proceedings, Max but to answer your last paragraph, the guy himself has decided to re-question the results from these meetings (I would hardly call them "properly controlled, double blind tests). They weren't ABX tests, btw - you seem to confuse the two - ABX is a forced choice test where you are being asked to identify X as either A or B. The tests you refer to were simply to identify if there was a difference between A & B - which should be a lot simpler, I know. Why no one found any difference between any DACs is something to be investigated which I believe he & others are attempting to do at another meeting planned for September?

 

[maxflinn]

John, I'm aware of what ABX testing is but perhaps that isn't what Vital hosted the first time, I thought it was.

 

[jkeny]

I think what has been lacking up to now is the encouragement that such tests can be passed. We had biased listeners so much that at the first sign of difficulty, they would give up with a negative outcome. I suspect from here on, people will try harder and differences that do exist will come out.

I believe you are correct, Amir & that was the point I was making. This whole area of bias removal is such a minefield of contention - the people on forums usually calling for (heckling, more like) blind tests often only recognise sightedness as the bias to be controlled & every other bias & factor ignored - it really is no wonder that most such tests return a null result in identifying subtle differences between DACs, for instance.

Orb has said it many times - it is not easy to correctly setup & run a DBT

 

[Ashley James]

I posted this in a other thread and it seems appropriate to put it here as well.

On the subject of ABX testing. I think I was first involved in them when I was with ATC in the eighties and since then, whenever we need to know if differences exist, especially subtle ones, we have found they're totally reliable. They always identify differences and which is best. Absolutely no doubt whatsoever or we wouldn't keep using them for ours and no one else's benefit.

They also prove to anyone who has proper experience of them, that lots of differences they think they can hear they can't and that protracted listening periods are not as reliable. Not surprisingly audiophiles don't like ABX or anything else that finds them out.

Another thing ABXs reveal is that younger people can hear differences that those over say thirty five can't. This is probably because by that age you've lost about 10dB of hearing sensitivity at 3kHz and above. This is the equivalent of half the volume on the bit of the spectrum that gives clarity or intelligibility. It may also explain why thirty five is the age that many used to consider hi fi, no longer though because headphones have more or less killed it.

Most of our comparisons are between speakers where differences are easy to hear and ABX difficult to apply, but for people who believe op-amps or DACs sound different, they're perfect.

 

[maxflinn]

Well, I observed no differences between the 20 pass file v original when sighted and nor did anyone else, (I think?) so I wasn't expecting to blind either, (negative bias?) yet I aced it that way.

 

[esldude]

Well, I observed no differences between the 20 pass file v original when sighted and nor did anyone else, (I think?) so I wasn't expecting to blind either, (negative bias?) yet I aced it that way.

That experience isn't uncommon actually in blind testing. You hear something you don't even think sounds different, but find you can discern some difference you aren't aware of. Lots who have not done any of this tend to think it is almost always the reverse. Hearing something you know is different sighted and failing to find it blind.

 

[maxflinn]

That experience isn't uncommon actually in blind testing. You hear something you don't even think sounds different, but find you can discern some difference you aren't aware of.

Yes I imagine it is common, though it surprised me.

Lots who have not done any of this tend to think it is almost always the reverse. Hearing something you know is different sighted and failing to find it blind.

Indeed, and when what's found sighted isn't found blind, it's usually the blind testing that's put under scrutiny, not the sighted

 

[Ethan Winer]

Many people can hear the degradation after 20 passes, and Amir showed he could hear it after only five passes. This does not surprise me. But I used multiple passes only to make the degradation more obvious. Nobody listens or uses converters this way! If the goal is to determine how much money one needs to spend on a DAC - and I think it should be! - then testing a single generation makes far more sense. That's what really matters. More to the point, that test was done using an old SoundBlaster sound card that cost $25. Few serious audiophiles are using or considering buying stuff like that. I'm much more interested in learning how many people can pick out the original versus one generation of the more typical "high end" converter I used in this follow-up test:

Converter Loop-Back Tests

Anyone who believes modern consumer grade digital converters are lousy is welcome to email me your file choices. Note that the $250 converter I used in this second test has multiple input and output channels. The same basic converter is available in stereo for only $150.

--Ethan

 

[jkeny]

But, from my reading & research, there are aspects of sound that aren't amenable to quick A/B testing - some aspects of the soundfield we need time to accommodate to which means that long term listening (blind or otherwise) is the only real way to check for differences.

Things like solidity of the soundstage, ambience of the venue (assuming a recording of a live event) are all things that take time to evaluate & accommodate to. The area of study known as Auditory Scene Analysis is concerned with these important aspects of our perception of hearing - not just the mechanics of our ears (which are tested by rapid A/B switching) but what happens within our processing system to make sense of the sound waves impinging on the ears.

So, yes, ABX testing has it's strengths, just like measurements have their strengths & long term listening has its strengths - there is no ONE way that provides all the answers, I believe.

 

[maxflinn]

I reckon I could have listened (sighted) to both those files till the end of time and not noticed a difference, but one ABX test and....

 

[Ashley James]

As I said Jk, ABX or AB is absolute and you hear which is best. However the truth is that it's some time since the difference between competent DACs was audible or worth bothering to compare.

We bought three AV processors a couple of years ago, a Denon professional one, an expensive Rotel AV Pre only and a Yamaha RVX 667. There wasn't any audible difference between the digital bit of any of them, the Rotel had the worst OS and Yamaha's power amps weren't marvellous. This year even their power amps are superb.

Now I doubt anyone on this forum would hear any difference between an iPhone, a Weiss or a Benchmark in a properly conducted test, but there'd be a country mile between B & W, Focal, MA and Spendor and everyone would hear that easily.

And that's audiophiles for you.

 

[maxflinn]

Many people can hear the degradation after 20 passes, and Amir showed he could hear it after only five passes. This does not surprise me. But I used multiple passes only to make the degradation more obvious. Nobody listens or uses converters this way! If the goal is to determine how much money one needs to spend on a DAC - and I think it should be! - then testing a single generation makes far more sense. That's what really matters. More to the point, that test was done using an old SoundBlaster sound card that cost $25. Few serious audiophiles are using or considering buying stuff like that. I'm much more interested in learning how many people can pick out the original versus one generation of the more typical "high end" converter I used in this follow-up test:

Converter Loop-Back Tests

Anyone who believes modern consumer grade digital converters are lousy is welcome to email me your file choices. Note that the $250 converter I used in this second test has multiple input and output channels. The same basic converter is available in stereo for only $150.

--Ethan

That all makes sense to me, Ethan. It's a pity that nobody tried the second test on PFM, just to put the performance (albeit in a non-real world scenario) of the first sound card into perspective.