DIscussion of ABX results of Winer's Loopback files

[jkeny]

JK

You really need to set up a proper AB or ABX and hear for yourself, because this is going nowhere and it's not good for a chap to get so stuck into rubbishing a methodology he hasn't used. Otherwise you'd realise ABX or just AB is infallible, but I've only had about thirty years of it, so what would I know.

Ashley, I've posted my ABX results, have you? You never answered what control you used in your tests, did you? All you have demonstrated is your use of flawed AB testing for the last 30 years, I'm afraid. There are some well understood & published standards to adhere to when testing for subtle audio differences - I'm sure you could tell us which you used & how you applied the recommendations contained in those standards?

 

[jkeny]

Lunchbreak over! Thanks for the entertainment.

 

[Phelonious Ponk]

Best I can do, Tim is quote the father of Auditory Scene Analysis, Al Bregman
The detail behind this everyday,common perceptual function is interesting & enlightening


Sorry?? Exactly what are you saying, Tim - less subtle differences can be easily detected by ABX? Sure, who's denying that? They can also be detected by sighted listening, so what?

At least we agree on that much. My apologies for the misunderstanding. I read this...

You also are overlooking the fact that perhaps many people do not have the discipline or indeed capability to train themselves to isolate a difference within/between two sound streams so that they can then do successful ABXing. For them long term listening is essential

And assumed you were taking the position that Amir's analytical approach is required for ABX to be effective. It's not; it is only necessary (and who knows, for someone out there it may not be) when comparing subtlety on the brink of inaudibility. And while I'm sure less subtle differences than the examples we're talking about here can be detected through sighted listening, they can also be quite easily denied in sighted listening, as many things that are not there at all can be imagined during sighted listening. We firmly disagree on the relative merits of sighted listening vs. casual blind listening.

Thanks for the link. I'll look into Al Bergman and see if I can sort out how the analysis of a "process in which the auditory system takes the mixture of sound that it derives from a complex natural environment and sorts it into packages of acoustic evidence in which each package probably has arisen from a single source of sound." is relevant to the reproduction of 2-channel recordings.

Tim

 

[Whatmore]

Sorry you are correct, Whatmore - I was overlooking the irony of using systems that obviously audibly mask the measured & audible differences between DACs

Walks like a DAC, swims like a DAC, floats like a DAC, sells DACs
A witch!

But you've missed it again and I'm bored now

 

[maxflinn]

Neither am I. "Here's A. Here's B. Here's X. Is X A or B?" Actually puts a potentially damaging lag in time between A and X. The ability to switch immediately back and forth between A and B may actually be more effective. But demanding strict (and undefined) scientific methodology and controls from blind listening, while arguing points and positions based on completely uncontrolled sighted listening is common in this hobby, and not to be taken too seriously.

Hi Tim, I hope you're well and the band is still rockin

Good point there re A/B vs ABX.

I not only agree, I think this is obvious. But it is not to be taken seriously. Like the double standard above, it is just a weak excuse to dismiss the results.

Tim

Yes seemingly, alas

 

[maxflinn]

The results shown here for ABX give actual trial results & statistical confidence levels. Can you show me similar results from forum organised blind tests (remember, that what we are talking about)?

Oh dear, therein lies the problem of forum organised "blind tests"! I'll repeat again - they are no less anecdotal than sighted tests!!

Sure, remove sightedness but don't assume that's everything dealt with for assuring valid results

If, for example, you ignore the concept of embedding controls in the test you are just flying blind & the results are meaningless - it's the equivalent of doing measurements with uncalibrated equipment.

John, I think you're over-complicating things and you seem to be only lending credence to the results of blind-tests that suit your point of view while defining what kind of tests are valid and what aren't, based on logic only you seem to grasp.

I'm not sure either of us want a repeat of the protracted nonsense that ensued elsewhere lately so I'm happy to agree to disagree and leave it there.

 

[Phelonious Ponk]

Hi Tim, I hope you're well and the band is still rockin

Good point there re A/B vs ABX.





Yes seemingly, alas

Thanks Max. The band is a bit quiet at the moment, but that's a good thing. It gives us time to work on new material and work in the new bass player. Besides. It takes ungodly powerful air-conditioning to make rock-n-roll comfortable in North Carolina in August, and outdoor gigs, even on relatively cool summer nights are still sticky. July and August just aren't pleasant here, period. The winters, on the other hand, are Godsend. And Spring and Fall are both just gorgeous.

Tim

 

[jkeny]

John, I think you're over-complicating things and you seem to be only lending credence to the results of blind-tests that suit your point of view while defining what kind of tests are valid and what aren't, based on logic only you seem to grasp.

Max, my point is not really complicated at all. I've no problem in accepting the typical forum run blind test get-togethers as anecdotal evidence, just like sighted tests are.

To attempt to raise it to the level of scientific proof is where the complexity arises but, hey, that's the scientific method for you - what can you do except live with it. To ensure this rigour in a scientific test requires significant planning & implementation of many controls. This is well trodden ground & a number of standards documents were published to cover the factors that need to be controlled. So I think all would agree that this is the standard by which to conduct rigorous blind testing.

Now, if all factors aren't being dealt with then we have uncontrolled variables which demotes any such tests to the level of anecdotal evidence.

What is being argued here is "my anecdotal evidence is better than your anecdotal evidence" because I have dealt with one or two of the many factors needed. Well I don't see the evidence to draw this conclusion.

If I was to present measurements here that I had taken which showed that all speakers measure the same you would justifiably ask me about my measurement system & technique. Maybe my microphone was shot, maybe my ADC was insensitive, maybe background noise swamped the differences, maybe ..........there are many other factors. If I then repeated the test in a quieter environment but didn't check my ADC or microphone & still presented the same results - all speakers measure the same, this wouldn't validate the results, would it? This is what is happening here - the big, obvious factor, sightedness is being dealt with, volume matching is being dealt with but no other factors which can strongly influence the result are given consideration. A simple way of proving my measurements would be to measure a known control to the expected measurements & if my measurement system showed that indeed I'm getting the correct measurement for it then some assurance of the capability of the equipment is shown.

Controls of this nature are seldom if ever used in forum run blind tests. Indeed when I asked Ashley what controls he used he demured.

Such is the nature of tests in forums - they are anecdotal, no problem there but let's be democratic about it - it applies to all forum run blind test

So, now we get onto the ABX tests results. Unfortunately, you guys have pinned your hats to ABX testing & it has produced positive evidence that there is indeed a difference. The reasons for this are being debated currently

If you don't accept these results then you have to state the shortcomings of the ABX test (that has always been championed)

So yea, I'm happy to agree to disagree after stating my position above.

 

[Ashley James]

Anyone would think we'd asked you to give Kim Jong Un a kidney!

You really need to get out more.

 

[maxflinn]

John, I don't think anyone is disputing that the ABX test results posted by several people across several forums prove that audible differences do exist between Ethan's files.

Could you list the variables you think Vital's first blind A/B test didn't cover that the ones of Ethan's files did?

 

[jkeny]

Anyone would think we'd asked you to give Kim Jong Un a kidney!

You really need to get out more.

Thanks for the ad hom, Ashley.
Care to deal with anything I said?

 

[Ethan Winer]

Oops, Ethan, Amir has just passed your new 1 pass loopback test as he did your old 1 pass loopback test!!

How could you know which of the four Wave files Amir compared? How do you know he didn't identify the original versus 10th generation?

--Ethan

 

[jkeny]

John, I don't think anyone is disputing that the ABX test results posted by several people across several forums prove that audible differences do exist between Ethan's files.

Could you list the variables you think Vital's first blind A/B test didn't cover that the ones of Ethan's files did?

No, I wouldn't care to talk about a test that nobody here knows the details of.
And furthermore, I'm not intending to do the research for you about how to run rigorous blind tests. If you "believe" in them - then the onus is on you to understand what you "believe in" & how to do them correctly.

 

[jkeny]

How could you know which of the four Wave files Amir compared? How do you know he didn't identify the original versus 10th generation?

--Ethan

I'll let yourself & Amir work that out - I see now that I misspoke?

 

[maxflinn]

No, I wouldn't care to talk about a test that nobody here knows the details of.

Fair enough. I PM'd Vital a link to this thread, perhaps he'll join in and fill us in.

And furthermore, I'm not intending to do the research for you about how to run rigorous blind tests. If you "believe" in them - then the onus is on you to understand what you "believe in" & how to do them correctly.

Like how you've flipped that

As I said before, I'm aware of what blind testing is about and, while most rigorous is best, just blind and level matched is often enough for the differences to disappear.

But those who 'believe' in differences, for whatever reasons, will 'believe' that they were there regardless...

 

[jkeny]

As I said before, I'm aware of what blind testing is about and, while most rigorous is best, just blind and level matched is often enough for the differences to disappear.

How do you know that some other uncontrolled factor(s) aren't responsible for returning the null result?

I know it sounds very simple, Max - eliminate the sighted bias & if it returns no difference then it was sightedness that was causing the difference heard, & not that it is ACTUALLY a difference. But you have to understand that we are dealing with perception of subtle differences - it ain't that easy & as I said, it's a well trodden area of experimentation - the standards were well established before & during the testing of codecs.

When they were checking how audible various codecs were, why didn't they just remove sightedness & publish their results?

 

[maxflinn]

How do you know that some other uncontrolled factor isn't returning the null result?

We don't, but then why would some other uncontrolled factor not cause the same null result when sighted? Is the process not one of elimination to some extent?

Differences present sighted.

Remove the variable that is sightedness.

Differences gone.

Conclude???

I know it sounds very simple, Max - eliminate the sighted bias & if it returns no difference then it was sightedness that was causing the difference heard, & not that it is ACTUALLY a difference. But you have to understand that we are dealing with perception of subtle differences - it ain't that easy & as I said, it's a well trodden area of experimentation - the standards were well established before & during the testing of codecs.

John, there seems to be broad agreement on this thread that to differentiate the smallest of differences, as Tim says 'those that are on the edge of audibility', one should use proper A/B or ABX protocols.

I'm not saying that's simple, but you don't need to hire NASA!

 

[FrantzM]

We don't, but then why would some other uncontrolled factor not cause the same null result when sighted? Is the process not one of elimination to some extent?

Differences present sighted.

Remove the variable that is sightedness.

Differences gone.

Conclude???


<snip>!

Can I use and edit that part of your post ever so slightly?

Differences and gross differences at that with knowledge of the components under test
Remove the knowledge.
Differences gone or too often differences perceived when there aren't any (same component playing but mistaken for different ones)

Conclude???

 

[maxflinn]

Can I use and edit that part of your post ever so slightly?

Certainly sir

Differences and gross differences at that with knowledge of the components under test
Remove the knowledge.
Differences gone or too often differences perceived when there aren't any (same component playing but mistaken for different ones)

Conclude???

Expectation bias?

EDIT - Sorry I'm a bit slow tonight and was answering my own question there, wasn't I

 

[jkeny]

We don't, but then why would some other uncontrolled factor not cause the same null result when sighted? Is the process not one of elimination to some extent?

Differences present sighted.

Remove the variable that is sightedness.

Differences gone.

Conclude???

Conclusion - that sightedness brings a bias to what we hear. It doesn't prove that there is actually no difference
What you have to grasp is that testing for perceptually subtle differences is difficult.

The usual argument is that declarations of night & day differences is hyperbole, Absolutely, agreed - these are subtle differences.