ABX tests - required or rejected?

Rodney Gold · Dec 9, 2017

It not possible to easily instantly and seamlessly AB most anything hifi ..
How would you AB an amp or a pre , or speakers or an arm , or tt etc seamlessly and instantly...?
You gonna have to jump thru hoops to set it up and it is already flawed in any objectivists mind anyway...

JackD201 · Dec 9, 2017

I guess that is exactly the point. If you aren't trying to convince anybody why go for it for anything but fun or torture depending on your perspective?

Just go sighted and take your time. Time is our friend. Go through the paces and that new gear crush will subside. Go through more paces and you'll get a better idea of what it does within your context. After all, it's that crush we all have to be careful about.

The "paces" are what I think more people should spend more time developing. It has to be a routine that is practiced with some degree of discipline. Line up the test tracks, edit some loops for particular performance parameters, test the devices lower and upper limits and not just nominal levels, see if system adjustments can deal with anomalies. If not, it's likely an intrinsic trait. It's not blind but at least a routine is dispassionate. Hopefully at the end of it all, there's enough data to make decisions based on merit rather than whatever hype we're afraid of falling for in the first place. To me this is the middle ground.

sbo6 · Dec 9, 2017

Exactly, there is no way to officially AB. But you could employ some aspects like JackD201 stated above.

Ron Resnick · Dec 9, 2017

microstrip said:
Did you ever carry a proper A/B/X test?

No; my assertion in post #17 is that it is very difficult, if not impossible, to carry out a "proper" A/B/X test.

Ron Resnick · Dec 9, 2017

microstrip said:
You are mixing A/B and A/B/X (the OP subject), and even now addressing compromised A/B comparisons. Are you mainly addressing the short term / long term listening subject?

I intended to address A/B comparisons using our reasonable best efforts to stay intellectually honest versus long-term listening of a single component without any effort at all to compare components contemporaneously.

microstrip · Dec 9, 2017

Ron Resnick said:
I intended to address A/B comparisons using our reasonable best efforts to stay intellectually honest versus long-term listening of a single component without any effort at all to compare components contemporaneously.

Ron,

IMHO if we want to stay intellectually honest we must carry the proper research on the subject. As soon as we read the basics we find that unfortunately the "reasonable best efforts" introduce uncontrolled factors that can invalidate our conclusions.

BTW, the first think we should stay clearly what type of equipment and differences we want to compare. Are you focusing in large and easily detectable impairments or the so called "small differences", that have no measured correlation with sound quality? What are you exactly meaning for "contemporaneously"?

Empirical Audio · Dec 9, 2017

Ron Resnick said:
These are excellent questions sbo6. Peter A and I were struggling with these very questions earlier today!

For me the question is not A/B/X versus measurements (I am a subjectivist so I reject the measurements-only view); for me the question is A/B/X, or at least A/B, versus long-term listening.

Both methodologies have their partisans, and both methodologies have significant, obvious flaws and biases and issues. My personal view is that I think there are more issues with the long-term listening approach than with the A/B/X approach. My personal view is that if one cannot reliably detect a difference using the A/B/X approach or at least the A/B comparison, then either there is no difference or the difference is too small to matter.

See http://www.whatsbestforum.com/showthread.php?22972-Comparative-Listening-Tests

This I agree with (long-term listening). It's really easy for people to go down the garden path with their own systems. I even catch myself once in a while and have to pull back. Things like higher jitter or tube distortion can be "pretty" and lead you down the wrong path.

Steve N.
Empirical Audio

Empirical Audio · Dec 9, 2017

Ron Resnick said:
I intended to address A/B comparisons using our reasonable best efforts to stay intellectually honest versus long-term listening of a single component without any effort at all to compare components contemporaneously.

If I were, for instance, trying to convince a recording engineer that he needs better cables, or a better DAC, I think I would design the ABX to his satisfaction, with my input of course. If there is any doubt in the guys mind you are trying to convince, the results are moot. If designed with all of the OP's objectives met and with input as to which listeners etc. from me, I think ABX could be useful. Trained listeners are critical I have found.

Another case where ABX could be useful is where the measurements are insufficient. There are lots of these cases IME. Jitter and dynamic response are both in need of improved measurement technique and stimulus. How many reviews have you read in Stereophile where JA says the measurements are great and the reviewer gives a lackluster review? How about the crappy measurement results where the reviewer says the thing sounds amazing?

Steve N.

Folsom · Dec 15, 2017

This thread is sort of funny. There are a lot of claims about wanting a very objectionistly ran test, to determine subjectively determined results. In general I don't think there a forum that gets laughed at more by people that run objectionistly ran, thorough, tests that gauge subjective results, than this one. And I mean by miles and miles. There are countless people that laugh at things they read on this forum, because they do all these tests that "prove" everyone here is insane with more $ than brains.

I don't agree with them at all, even though I coincide with a lot of electronic development that they participate in.

For a manufacturer there are literally only two choice, and not one of them includes running a thoroughly vetted ABX, double blind test.
1. You design purely from a knowledge stand point and don't listen to it, basically an electronic engineer makes it and tries to incorporate some known knowledge, then checks measurements.
2. Someone listens, be it the engineer, a panel of people, doesn't matter. This can exist with #1, but is not consistently seen with #1.

A big roblem with ABX:

One of issues is that people try way too hard. For a designer of sorts you have to learn to go with instinct or use familiarity. Instincts are right more often than you think. When I ask non-audiophile people they often guess right but are too scared to even answer you, so they stew on it and try too hard, repeating and repeating the test. It goes no where.

It isn't abnormal for a designer to try and pick stuff by listening a little and then having some trusted ears try something out. Feedback is very important. When you measure something it can look great, and skew your impression of the performance. Then when someone hears it who doesn't know, doesn't care, they might tell you another story.

But anyways; you may have noticed I'm all about long term familiarity. For most people this is the only way to tell. Back to the big joke, is that everyone else on the internet is laughing at WBF because they know that almost no one here can tell the difference in sound in an ABX that is blind, no matter if it's done well or poorly. The only thing that ABX tests seem to tell us is that we can't get results from ABX tests. Fact of the matter is a lot of stuff that measure very differently can't even be detected in these types of tests. And yet if you replace someone's TotalDAC they've had for a year, with a Formula, they could tell from the other side of the house.

Ron Resnick · Dec 16, 2017

Folsom said:
This thread is sort of funny. There are a lot of claims about wanting a very objectionistly ran test, to determine subjectively determined results. In general I don't think there a forum that gets laughed at more by people that run objectionistly ran, thorough, tests that gauge subjective results, than this one. And I mean by miles and miles. There are countless people that laugh at things they read on this forum, because they do all these tests that "prove" everyone here is insane with more $ than brains.

I don't agree with them at all, even though I coincide with a lot of electronic development that they participate in.

For a manufacturer there are literally only two choice, and not one of them includes running a thoroughly vetted ABX, double blind test.
1. You design purely from a knowledge stand point and don't listen to it, basically an electronic engineer makes it and tries to incorporate some known knowledge, then checks measurements.
2. Someone listens, be it the engineer, a panel of people, doesn't matter. This can exist with #1, but is not consistently seen with #1.

A big roblem with ABX:

One of issues is that people try way too hard. For a designer of sorts you have to learn to go with instinct or use familiarity. Instincts are right more often than you think. When I ask non-audiophile people they often guess right but are too scared to even answer you, so they stew on it and try too hard, repeating and repeating the test. It goes no where.

It isn't abnormal for a designer to try and pick stuff by listening a little and then having some trusted ears try something out. Feedback is very important. When you measure something it can look great, and skew your impression of the performance. Then when someone hears it who doesn't know, doesn't care, they might tell you another story.

But anyways; you may have noticed I'm all about long term familiarity. For most people this is the only way to tell. Back to the big joke, is that everyone else on the internet is laughing at WBF because they know that almost no one here can tell the difference in sound in an ABX that is blind, no matter if it's done well or poorly. The only thing that ABX tests seem to tell us is that we can't get results from ABX tests. Fact of the matter is a lot of stuff that measure very differently can't even be detected in these types of tests. And yet if you replace someone's TotalDAC they've had for a year, with a Formula, they could tell from the other side of the house.

How would we know if they can tell from the other side of the house -- or merely if they think they can tell from the other side of the house?

microstrip · Dec 16, 2017

Folsom said:
This thread is sort of funny. There are a lot of claims about wanting a very objectionistly ran test, to determine subjectively determined results. In general I don't think there a forum that gets laughed at more by people that run objectionistly ran, thorough, tests that gauge subjective results, than this one. And I mean by miles and miles. There are countless people that laugh at things they read on this forum, because they do all these tests that "prove" everyone here is insane with more $ than brains. (...)

I do not find this thread funny, although it is just more of the same old arguments, nothing really new. Curiously I do not see any relevant high-end audiophile laughing at the content of WBF, except on humorist posts. Also I have never seen an amateur properly carried blind "test" involving high-end equipment in the net. Surely, YMMV.

thedudeabides · Dec 16, 2017

the sound of Tao said:
I feel that objective assessment is useful to navigate is to the right post code but long term subjective assessment can actually get us to a more exact destination. What jumps out as better in short term can actually just be about more obvious elements of the sound. Long term musical satisfaction is not necessarily immediately obvious.

At a time when agood mate had both Verity Parsifals and his Tune Animas going from the Animas to the Veritys all the strengths of the Parsifals jumped out... ah yes, that confidence and control and it was offering up a different spectrum of sonic strengths. It wasn’t till you played them for quite some time that you realised for all their sonic prowess that they just didn’t do music the same way and that something less tangible was missing.

It’s way too easy to get caught up in the impressive and obvious parts when doing ABX and not then also assess for the musical whole which is easier to appreciate when you aren’t focussing and analysing. Good whole assessment isn’t just about the obvious I figure.

+1. I've always believed that taking this type of test is similar to watching a "movie promo trailer" and then judging the movie absent watching it in its entirety.

jkeny · Dec 20, 2017

Ron Resnick said:
I think that formal scientific protocol is not remotely plausible for our hobbyist A/B comparison testing. We are stuck just trying to be as honest with ourselves and our comparisons as is reasonably practicable.

So the answer is not that A/B comparisons are worthless because they don't satisfy formal scientific protocol. The question is whether our compromised A/B comparisons are better than (in my opinion) even more compromised and flawed long-term listening "tests" with no direct, contemporaneous comparisons.

And how could you possibly judge this differential without knowing the flaws in each listening method?
You don't have any measure of the number of flawed ABX tests (which are the test of choice for those demanding 'proof') being reported as perfectly valid null results.
The lack of controls in ABX tests rule out this possibility - without controls, the ABX test itself is flawed

Ron Resnick · Dec 20, 2017

You are arguing that A/B/X tests are flawed. But don't you see that I already stipulated that?

I also stipulated that long-term listening is flawed. I clearly concede that both methods have biases and issues and that both methods are compromises.

My conclusion is that in my personal opinion I think A/B or A/B/X is less compromised than long-term listening. Respectfully, I do not understand what you are arguing about.

What are the "controls" in the long-term listening method?

the sound of Tao · Dec 20, 2017

thedudeabides said:
+1. I've always believed that taking this type of test is similar to watching a "movie promo trailer" and then judging the movie absent watching it in its entirety.

Great analogy, it’s easy to identify the more obvious aspects of a movie from a short snapshot but completely miss the meaning of the experience. Time is an attribute of truer and deeper understanding.

When you are sitting there actively looking for an awareness or an appreciation it may well be the very act of focussed observation that holds you back from the changed state of true realisations.

How often has an awareness just grabbed your attention when you weren’t at all trying to make it do just that. Timing is everything, realisation comes only when it is ready.

There is just so much grasping in consciousness. Music leads us beyond limits, well past enlightenment, knowing is not temporal nor spatial, conscious awareness is rarely the end, just a phase to be let go of. ABX me nothing.

jkeny · Dec 21, 2017

Ron Resnick said:
You are arguing that A/B/X tests are flawed. But don't you see that I already stipulated that?

I also stipulated that long-term listening is flawed. I clearly concede that both methods have biases and issues and that both methods are compromises.

My conclusion is that in my personal opinion I think A/B or A/B/X is less compromised than long-term listening. Respectfully, I do not understand what you are arguing about.

What are the "controls" in the long-term listening method?

Yes, there are no controls in either method - that's the point

I understand your point but you are opining that one listening method is superior to another with absolutely no basis on which to make this judgement. My reading is that you posed your opinion as a rhetorical question in its framing & I'm giving you the opinion that there can be no evaluation between methods based on the fact that there are no error statistics for each.

People get carried away with terms & assume that if it sounds scientific it must be better.

So let's admit that both are flawed. Here's the difference as I see it - one method is prone to giving false positives & one method is prone to deliver false negatives. We have no measure of the number of such falsities on either side so no evaluation can be made about relative superiority. All we can do is take a view on is which error we can better live with. That is a choice which I believe involves so many other traits in our personalities that it's not a worthwhile debate.

Relying exclusively on one listening method seems to me to be flawed

audioguy · Dec 21, 2017

Ron Resnick said:
For me the question is not A/B/X versus measurements (I am a subjectivist so I reject the measurements-only view); for me the question is A/B/X, or at least A/B, versus long-term listening.

Those methods are not mutually exclusive. I have participated in A/B (not A/B/X) tests where I could listen as long as I wanted - minutes, hours or days. I've done so with cables and pre-amps and amplifiers. I don't know if A/B blind tests are perfect but I do know, without a doubt, that I am not able to exclude expectation bias, so non-blind just doesn't work for me. And I am equally sure that many others are not immune to expectation bias (but are not willing to admit it).

I've gotten to the point where tiny/minuscule differences/improvements (?) are no longer critical to my listening pleasure. If I can't hear them in a blind test without straining, I'm not interested in spending my money. I have not always been that way - and my current approach is a lot less expensive -- a lot less

microstrip · Dec 21, 2017

audioguy said:
Those methods are not mutually exclusive. I have participated in A/B (not A/B/X) tests where I could listen as long as I wanted - minutes, hours or days. I've done so with cables and pre-amps and amplifiers. I don't know if A/B blind tests are perfect but I do know, without a doubt, that I am not able to exclude expectation bias, so non-blind just doesn't work for me. And I am equally sure that many others are not immune to expectation bias (but are not willing to admit it).

I've gotten to the point where tiny/minuscule differences/improvements (?) are no longer critical to my listening pleasure. If I can't hear them in a blind test without straining, I'm not interested in spending my money. I have not always been that way - and my current approach is a lot less expensive -- a lot less

I was preparing a post on a similar line - A/B or A/B/X are just methods of comparative listening, that can be carried independently of time. Did you get results that were statistically valid in your blind tests?

audioguy · Dec 21, 2017

microstrip said:
I was preparing a post on a similar line - A/B or A/B/X are just methods of comparative listening, that can be carried independently of time. Did you get results that were statistically valid in your blind tests?

They were very statiscally valid - and clearly demonstrated I could not hear a difference in most cases. The lone exception was a comparison between a new $9000 Levinson amp (don’t recall the model number) and a used $700 Bryston 4B. The Bryston was the far superior amp for my speakers (Dunlavy SC-VI’s). Nothing subtle about the differences - and they were level matched. And not at all surprising was that prior to blind testing when listening sited, the shiny new and cool looking Levinson sounded better than the beat up Bryston!

Gregadd · Dec 21, 2017

It has been argued that rapid switching between short clips, is not only okay, but preferrred.

ABX tests - required or rejected?

Member

WBF Founding Member

VIP/Donor

Site Co-Owner, Administrator

Site Co-Owner, Administrator

VIP/Donor

Industry Expert

Industry Expert

VIP/Donor

Site Co-Owner, Administrator

VIP/Donor

Well-Known Member

Industry Expert, Member Sponsor

Site Co-Owner, Administrator

Well-Known Member

Industry Expert, Member Sponsor

WBF Founding Member

VIP/Donor

WBF Founding Member

WBF Founding Member

Similar threads