DIscussion of ABX results of Winer's Loopback files

jkeny · Jul 30, 2014

Ashley James said:
Jk

You're beating this one to death and appear to be trying desperately to discredit a well known, well tried, approved and tested method of comparison. There is no better way and no amount of obfuscation will change that. As I explained, we were doing it for our own benefit and we did it often with different discriminators in FM Tuners, various amplifier circuits, op-amps, all sorts of things to make sure we did the best possible job.

Sorry, Ashley, if you don't follow correct procedures & implement correct controls in blind testing, you are very likely to return a null result, as you did. A null result is not proof of anything, as you should know. Using such a result, particularly when the test is flawed, to make bold statements about the lack of differences between DACs is doubly flawed. No amount of appeal to tradition with "well known, well tried, well approved and tested" will work - if you don't know how to carry out listening tests for differentiating subtle audible differences then your tests are worthless, I'm afraid.

However things have moved on and all the money on development has gone into phones and their DACs or AV. The headphone business is about ten times the size of hi fi and headphones are more revealing. This hasn't stopped the relentless march of progress in AV though and now a £500 Yamaha processor is not only as good a DAC as you'll get, but despite all being crammed in together, you won't hear better powers amps either.

Indeed

Modern mass market electronics are largely as transparent as they need be, so all that's left is speakers and mostly they're pretty grim and losing out to the better soundbars and inexpensive AV efforts from the big Japanese companies. They're almost as good, a fraction of the price and they work seamlessly with the TV that's at the centre of all modern media systems.

As I said if you want to put down that million dollar challenge that you already mentioned, to differentiate a iphone from a well regarded DAC, please let us know.

Alan Sircom understands all this better than I do and on PF he's explained that legacy hi fi is a dead duck and how hard it is to write for such a small group who can't agree on anything. Or write reviews that satisfy both advertisers and readers.

Anyway my advice to you is to stop beating a dead and decomposing horse, get yourself an iPhone, buy some "mastered for iTunes" tracks and start with the standard ear buds. If you still think an ABX is necessary, you can tell us all and I'll have a quiet chuckle.

Oh, please, Ashley, your trolling is too transparent. I know both you & Winer were broth here by Maxflinn, as he has had a bad time on PinkFishMedia forum & blames this on me but please don't insult the intelligence of readers here.

jkeny · Jul 30, 2014

I already said, Tim - one example - how best the illusion is created using one system Vs another is one of the most important aspects of our listening to 2 channel stereo. It is determined by how well the audible cues are rendered for grouping of the different streams within the music. This is a dynamic process & runs over a longer period than quick A/B or ABX testing allows. Any anomalies within this rendering of the stream will kick the listener out of the illusion. Anomalies can only be judged by listening to the flow - they are not something that stick out as an audible Frequency or level differences between two systems.

I already said that we also need time to listen in a relaxed manner to a system over an extended period to get a feel for all it's characteristics. This may involve A/Bing ABXing & casual listening over this period. It is less prone to error to do all listening over an extended period than to do one A/B test & declare a result. I'm sure many have the experience of mistakes made in the past doing this - not being able to live with the chosen system over the long term.

In all regards, I find long term listening to be far less error prone than short A/Bing or ABXing or DBTing - these are useful for isolating one particular aspect or difference in a system but not for determination of the full character of a system or device.

You also are overlooking the fact that perhaps many people do not have the discipline or indeed capability to train themselves to isolate a difference within/between two sound streams so that they can then do successful ABXing. For them long term listening is essential

Ashley James · Jul 30, 2014

JK

As I said, you're flogging a dead horse and no you will not be able to tell an iPhone (or its competitors) from a top quality DAC now, even the measurements will tell you that. Nearly all our customers are one time audiophiles who've sold high end to switch to us and Apple, so we get to compare them often. Not ABX, but I'm sure it'd not reveal anything because they measure so well. If you think about it, then it's obvious that this was bound to happen. Stereo DAC chips haven't changed much in years, they've just become more integrated, the analogue stage everyone places emphasis on is actually part of the chip, so that just leaves the filter. If Wolfson get to sell a few thousand stereo chipsets, they barely notice because they'll sell a few million ones for phones in the same time. Only difference is phones are a competitive market and performance, ease of use, cost and battery life are all crucial. Not surprising this is where DAC manufacturers spend their money and, as a result, the devices just keep improving, so now they don't measure quite as well as an audiophile DAC, but it's close enough for no one to hear a difference. But everyone can between B & W, MA, Spendor and Focal, which should tell you where to look for improvements.

Move on JK, you're a stuck record and this is a dead subject.

Ash

Ashley James · Jul 30, 2014

jkeny said:
I already said, Tim - one example - how best the illusion is created using one system Vs another is one of the most important aspects of our listening to 2 channel stereo. It is determined by how well the audible cues are rendered for grouping of the different streams within the music. This is a dynamic process & runs over a longer period than quick A/B or ABX testing allows. Any anomalies within this rendering of the stream will kick the listener out of the illusion. Anomalies can only be judged by listening to the flow - they are not something that stick out as an audible Frequency or level differences between two systems.

I already said that we also need time to listen in a relaxed manner to a system over an extended period to get a feel for all it's characteristics. This may involve A/Bing ABXing & casual listening over this period. It is less prone to error to do all listening over an extended period than to do one A/B test & declare a result. I'm sure many have the experience of mistakes made in the past doing this - not being able to live with the chosen system over the long term.

In all regards, I find long term listening to be far less error prone than short A/Bing or ABXing or DBTing - these are useful for isolating one particular aspect or difference in a system but not for determination of the full character of a system or device.

This tell me that you're hypothesising and have no actual experience of proper ABX. Otherwise you'd know it is as 100% reliable.

jkeny · Jul 30, 2014

Ashley James said:
JK

As I said, you're flogging a dead horse and no you will not be able to tell an iPhone (or its competitors) from a top quality DAC now, even the measurements will tell you that. Nearly all our customers are one time audiophiles who've sold high end to switch to us and Apple, so we get to compare them often.

You are telling me that your customers use iPhones or mobile phones as their DACs?

Not ABX, but I'm sure it'd not reveal anything because they measure so well. If you think about it, then it's obvious that this was bound to happen. Stereo DAC chips haven't changed much in years, they've just become more integrated, the analogue stage everyone places emphasis on is actually part of the chip, so that just leaves the filter.

Your glib way of talking about important aspects of DAC chips is revealing, Ashley - analogue stage is dealt with because it is now on chip & therefore perfect? Is that what you are saying?

If Wolfson get to sell a few thousand stereo chipsets, they barely notice because they'll sell a few million ones for phones in the same time. Only difference is phones are a competitive market and performance, ease of use, cost and battery life are all crucial. Not surprising this is where DAC manufacturers spend their money

I don't disagree with you there - it is where DAC manufacturers spend their money

and, as a result, the devices just keep improving,

Oh so they only pay attention to low voltage & current operation & this improves the sound? Can you tell us how this feat is achieved, please?

so now they don't measure quite as well as an audiophile DAC, but it's close enough for no one to hear a difference. But everyone can between B & W, MA, Spendor and Focal, which should tell you where to look for improvements.

I'm afraid this mantra is very similar to another trader on this thread who sells room treatment - "no other equipment is of importance, don't spend your money there, spend it here on my lovely speakers/room treatment"

Move on JK, you're a stuck record and this is a dead subject.

Ash

I believe this thread was about discussion of Winer's results until you & Maxflinn came along with Winer. You have shown us your opinion of the audio field which is revealing so unless you have something to say about the perfection of the soundcards which can be audibly identified with 1 pass loopback test then you have said all you wanted.

jkeny · Jul 30, 2014

Ashley James said:
This tell me that you're hypothesising and have no actual experience of proper ABX. Otherwise you'd know it is as 100% reliable.

Ashley, I'm not going to do the circular, tit-for-tat, no substance posting which you seem to want to engage in. Please desist from mantras & offer something of substance. Your avoidance in answering the blind test controls I asked you about that are necessary for any worthwhile blind testing, reveals more than all your bluster & mantra declarations

Whatmore · Jul 30, 2014

jkeny said:
I'm afraid this mantra is very similar to another trader on this thread who sells room treatment - "no other equipment is of importance, don't spend your money there, spend it here on my lovely speakers/room treatment"

I'm guessing the irony isn't lost on you here?

jkeny · Jul 30, 2014

Whatmore said:
I'm guessing the irony isn't lost on you here?

I have never denied that room treatment & speakers give more return on investment than money spent on the source. Clear enough?
I would never say that the speakers in your iPhone or equivalent are sufficient for high quality sound.
Some people just happen to believe that getting as pristine a source as possible also pays dividends as it can't be fixed further downstream.
It takes all sorts to make up the world & this hobby!

Ashley James · Jul 30, 2014

JK

1. You're clutching at straws. You haven't experience of ABX testing, if you had, we'd not be having this argument. You'd know that I was right and it works.

2. iPhones etc are good enough for good headphones that will be better and more revealing than any speaker you've experienced, so don't knock them.

3. I actually said that there are huge differences between speakers, so that's where to look for improvement rather than struggling to hear tiny if they exist differences between DACs. Obviously it's a given that I'd prefer people to buy mine, but I don't like beating them over the head. I'm sixty-eight and don't need to work, so I'm relaxed - are you?

Ashley James · Jul 30, 2014

I think insulting manufacturers is foolish, yet it's common on audiophile forums. it's perfectly obvious that any of them would prefer that you bought their products and if you're smart, you'll factor that out of what they say. Look at JK banging away, but also they are the experts and they're doing the research, so if you attack them you won't learn anything. You might show that they have weaknesses, but is that as helpful as acquiring more knowledge of a subject you're passionate about?

Having said all that, is there anyone on here who believes DACs sound different that thinks the differences are even a fraction of those between speakers, leave alone active speakers such as we make. I cannot believe there is.

jkeny · Jul 30, 2014

Ashley James said:
JK

1. You're clutching at straws. You haven't experience of ABX testing, if you had, we'd not be having this argument. You'd know that I was right and it works.

Like Max, you are mixing up ABX testing with the "blind test" you described (DAC evaluation boards). I have done Foobar ABX tests & the results are already on record. I haven't been involved in a properly conducted blind AB test & I bet few here have, barring Amir, J_J & possibly some others. I have done the type of test you have described for my own purposes as sanity checks but wouldn't dream of using it as "proof" of anything

2. iPhones etc are good enough for good headphones that will be better and more revealing than any speaker you've experienced, so don't knock them.

3. I actually said that there are huge differences between speakers, so that's where to look for improvement rather than struggling to hear tiny if they exist differences between DACs. Obviously it's a given that I'd prefer people to buy mine, but I don't like beating them over the head. I'm sixty-eight and don't need to work, so I'm relaxed - are you?

What am I 68 or am I relaxed? No & yes!
And I'm off to do some work - catch you later, perhaps?

The argument here is like asking on a car forum "who thinks the engine is the most important part of the car"? If you put bad fuel into it you will quickly realise that all elements play their part in the experience of driving.

Whatmore · Jul 30, 2014

jkeny said:
I have never denied that room treatment & speakers give more return on investment than money spent on the source. Clear enough?
I would never say that the speakers in your iPhone or equivalent are sufficient for high quality sound.
Some people just happen to believe that getting as pristine a source as possible also pays dividends as it can't be fixed further downstream.
It takes all sorts to make up the world & this hobby!

It seems it was lost on you after all

maxflinn · Jul 30, 2014

jkeny said:
Yes, weaknesses is a better term

Fair enough.

Again, you're confusing ABX with the usual forum blind A/B tests

I'm not so sure that the former is much better than the latter.

Yes, exactly - sure they can with difficulty but they seldom are catered for & this is why the tests are flawed

Vital's A/B tests seemed to cater for everything so I wouldn't call them flawed.

Sure, that's one of the checklist covered, perhaps?

Yes, it seems like good, sensible practice, to me.

Sighted testing is considered to be just anecdotes. So too are the blind tests without the necessary controls/factors catered for - just anecdotes. Trying to elevate them to better than sighted tests is just a religious belief & does a disservice to well organised blind testing.

Sighted testing is oft put forward as evidence though, think our friend Mark for example, not anecdotal, as it should be seen.

I disagree that only strictly controlled double-blind ABX testing is worthy of elevation above sighted testing as a means for discerning potential audible differences. Removing as many biases as possible is best, but removing some is better, the main one being the knowledge of what component or file is being tested at any given time, followed closely by level matching, and so on.

So, tempering this with our knowledge of just how poor audio memory is, I'd rate it like this - from worst to best -

Long-term sighted, non-level matched comparison and, any other form of long-term testing.

Short-term sighted, non-level matched.

Short-term sighted, level-matched.

Short-term blind, level matched A/B or ABX.

Phelonious Ponk · Jul 30, 2014

I already said, Tim - one example - how best the illusion is created using one system Vs another is one of the most important aspects of our listening to 2 channel stereo. It is determined by how well the audible cues are rendered for grouping of the different streams within the music. This is a dynamic process & runs over a longer period than quick A/B or ABX testing allows. Any anomalies within this rendering of the stream will kick the listener out of the illusion. Anomalies can only be judged by listening to the flow - they are not something that stick out as an audible Frequency or level differences between two systems.

Uh...yeah. OK. What cues? Define "different streams within the music?

You also are overlooking the fact that perhaps many people do not have the discipline or indeed capability to train themselves to isolate a difference within/between two sound streams so that they can then do successful ABXing. For them long term listening is essential

I'm not overlooking it, I'm dismissing it. I understand that what Amir's and others' results here have shown is that training, analysis and ABX testing can reveal differences so subtle that they are on the edge of audibility. I am not leaping to the conclusion that the training and methodology is required for successful ABXing. In fact, I know from experience that less subtle differences are very easily detected through ABX.

Tim

Orb · Jul 30, 2014

Phelonious Ponk said:
Uh...yeah. OK. What cues? Define "different streams within the music?

I'm not overlooking it, I'm dismissing it. I understand that what Amir's and others' results here have shown is that training, analysis and ABX testing can reveal differences so subtle that they are on the edge of audibility. I am not leaping to the conclusion that the training and methodology is required for successful ABXing. In fact, I know from experience that less subtle differences are very easily detected through ABX.

Tim

Tim,
whose ABX are you referring to?
Because in the past there used to be a hobbyist site that compared amps/dacs/think a couple of other things and the conclusion was listeners could not identify the products using ABX when carefully level matched; they had one or two ABX passes out of a lot of products tested.
The reality is sort of between the two, but if one fails to identify differences then it needs to go to the trained listening-analytical-methodical-specific approach.
ABX is just a tool to identify differences, so knowing what one is focusing on,where,trait is important in this context.
Otherwise might as well do level matched blind A/B and base it upon preference-like.
Thanks
Orb

Phelonious Ponk · Jul 30, 2014

JK -

Again, you're confusing ABX with the usual forum blind A/B tests

Maxxflinn --

I'm not so sure that the former is much better than the latter.

Neither am I. "Here's A. Here's B. Here's X. Is X A or B?" Actually puts a potentially damaging lag in time between A and X. The ability to switch immediately back and forth between A and B may actually be more effective. But demanding strict (and undefined) scientific methodology and controls from blind listening, while arguing points and positions based on completely uncontrolled sighted listening is common in this hobby, and not to be taken too seriously.

I disagree that only strictly controlled double-blind ABX testing is worthy of elevation above sighted testing as a means for discerning potential audible differences. Removing as many biases as possible is best, but removing some is better, the main one being the knowledge of what component or file is being tested at any given time, followed closely by level matching, and so on.

I not only agree, I think this is obvious. But it is not to be taken seriously. Like the double standard above, it is just a weak excuse to dismiss the results.

Tim

jkeny · Jul 30, 2014

Phelonious Ponk said:
Uh...yeah. OK. What cues? Define "different streams within the music?

Best I can do, Tim is quote the father of Auditory Scene Analysis, Al Bregman

The long-term aim of our research has been to understand "auditory scene analysis" (ASA), a process in which the auditory system takes the mixture of sound that it derives from a complex natural environment and sorts it into packages of acoustic evidence in which each package probably has arisen from a single source of sound. This grouping helps pattern recognition not to mix information from different sources.

The detail behind this everyday,common perceptual function is interesting & enlightening

I'm not overlooking it, I'm dismissing it. I understand that what Amir's and others' results here have shown is that training, analysis and ABX testing can reveal differences so subtle that they are on the edge of audibility. I am not leaping to the conclusion that the training and methodology is required for successful ABXing. In fact, I know from experience that less subtle differences are very easily detected through ABX.

Tim

Sorry?? Exactly what are you saying, Tim - less subtle differences can be easily detected by ABX? Sure, who's denying that? They can also be detected by sighted listening, so what?

Ashley James · Jul 30, 2014

JK

You really need to set up a proper AB or ABX and hear for yourself, because this is going nowhere and it's not good for a chap to get so stuck into rubbishing a methodology he hasn't used. Otherwise you'd realise ABX or just AB is infallible, but I've only had about thirty years of it, so what would I know.

jkeny · Jul 30, 2014

maxflinn said:
Fair enough.

I'm not so sure that the former is much better than the latter.

The results shown here for ABX give actual trial results & statistical confidence levels. Can you show me similar results from forum organised blind tests (remember, that what we are talking about)?

Vital's A/B tests seemed to cater for everything so I wouldn't call them flawed.

Oh dear, therein lies the problem of forum organised "blind tests"! I'll repeat again - they are no less anecdotal than sighted tests!!

Yes, it seems like good, sensible practice, to me.

Sure, remove sightedness but don't assume that's everything dealt with for assuring valid results

Sighted testing is oft put forward as evidence though, think our friend Mark for example, not anecdotal, as it should be seen.

I disagree that only strictly controlled double-blind ABX testing is worthy of elevation above sighted testing as a means for discerning potential audible differences. Removing as many biases as possible is best, but removing some is better, the main one being the knowledge of what component or file is being tested at any given time, followed closely by level matching, and so on.

If, for example, you ignore the concept of embedding controls in the test you are just flying blind & the results are meaningless - it's the equivalent of doing measurements with uncalibrated equipment.

jkeny · Jul 30, 2014

Whatmore said:
It seems it was lost on you after all

Sorry you are correct, Whatmore - I was overlooking the irony of using systems that obviously audibly mask the measured & audible differences between DACs

DIscussion of ABX results of Winer's Loopback files

Industry Expert, Member Sponsor

Industry Expert, Member Sponsor

New Member

New Member

Industry Expert, Member Sponsor

Industry Expert, Member Sponsor

Well-Known Member

Industry Expert, Member Sponsor

New Member

New Member

Industry Expert, Member Sponsor

Well-Known Member

New Member

New Member

New Member

New Member

Industry Expert, Member Sponsor

New Member

Industry Expert, Member Sponsor

Industry Expert, Member Sponsor

Similar threads