Do you guys Enjoy or Extremely Dislike A/B Testing?

[caesar]

Personally, I passionately hate it. I am so busy, and if something is within 5-10%, I would rather just turn the system on and play music instead of f*cking around with moving gear, switching out cables, trying different footers, etc., only so you can hear minor differences.

 

[Keith_W]

Most of the time, I don't care. I just want to listen to music. However - every now and then I come across something which I think sounds magnificent. Since it is integrated into a system which is not my own, I don't know how that particular piece of equipment is contributing to the sound - in which case, I would want to A/B it in my own system. Or bring my equipment over to A/B it. I have to say, I DO enjoy this process. It tells me that something is either worth spending money on, or it isn't.

This is why I upgrade so infrequently.

 

[GaryProtein]

I am a side by side, by side, by side, by side testing kind of guy.

I do that with EVERYTHING from cars to winter coats to coffees to silverware to equipment of all sorts.

I DON'T necessarily enjoy it. I hate shopping.

I do it to avoid buyer's remorse.

 

[DaveC]

It's part of the job. Enjoy is too strong a word but the result is always interesting.

 

[amirm]

I am highly motivated to them but the moment I think about doing it, the hating starts. Few things are more boring than doing these kinds of tests so I only do it when I am really, really motivated. Then I go to do it and hating starts.

 

[still-one]

Prior to purchasing I set my amps, pre-amps, CD player, power line products and cables in my set-up, A/B'd them against what I owned or other contenders before deciding which was better. Just about everything except my speakers.

 

[Fitzcaraldo215]

To me, it is not a question of liking it. It simply must be done if you are truly intent on making a meaningful comparison. I do not enjoy it, and it can become tedious, especially for items that differ little, if at all, like cables, which require many swaps, reconnects, etc. But, I always learn something valuable from the experience, even if that is sometimes only that there is no difference worth getting excited about.

If the alternative is to skip side by side A-B in favor of "long term listening" to a single setup for days or weeks, I honestly do not see how I, or anyone, would really know what sonic differences they are dealing with. I am not against longer term listening with a wider range of recordings. But, I think that is typically useful only for confirmation after careful A-B.

Listening to equipment, whether A-B or longer term, to me is quite different from and with an entirely different focus than listening for musical enjoyment. I want to simply enjoy the music in better sound in the end. That is my goal. Equipment comparisons are an investment in potentially better sound, but a "necessary evil" to get there.

 

[dallasjustice]

I enjoy reading about others doing it. I hate participating in shootouts. The last shootout was between an "upgraded" MSB DAC and the "old" version. I forget all the stupid names. I told my friend that the "old" version sounded better to me.

 

[thedudeabides]

In all honesty, who would actually like doing this assuming they have a real life?

Another potential S v O thread it seems.

 

[Fiddle Faddle]

I despise A/B testing but I realise it is a very necessary part of the hobby if one is to achieve long-term benefits. Why do I despise it? For a few reasons. One is that it is incredibly time consuming. For me, it is never, ever enough to just pick a handful of tracks to use as comparisons. I need to use about 50 different tracks. Why? Because in my long experience, if you do not use a very large variety of tracks with completely different characteristics, you may just end up making a sideways move or even backward one. It might sound better on those few tracks you've chosen, but 20 of them might sound worse and another 25 might be no better. So my rule is that the vast majority of a large selection of tracks must sound better, a few can sound no better, but only 1 or 2 are allowed to sound worse (yes, this happens too but it is because the improvements reveal shortcomings that were previously glossed over or compensated for).

Another reason is that the testing is stressful. I don't find it the least bit fun as I am concentrating as much as I would in a game of chess. So my mind is stuffed after a few hours.

And finally - perhaps the hardest bit about it of all. Getting a consistent result. I have found that the testing procedure needs to repeated a few times, perhaps over a few days in a week to make sure the choices made always provide a consistent result. In some cases, for example, something as simple as being in a bad mood, having taken a prescribed drug or being tired, etc, can alter the perception as to whether something really is better or not. But I've found that it's usually possible with experience to make progress after about three, two-hour sessions held over a week (the "off" days to keep the ears fresh).

Just to give people an example - I have been doing a project over the last 4 years of transcribing my vinyl collection to CD. My transcription masters are 24/96 so I had to chose the appropriate resampling and dithering products and settings for CD production. I spent about 12 months coming up with the chosen products and settings purely for the CD production part of the project. This involved around 200 x A/B listening sessions and 60 burned CDs. The A/B sessions were required to "narrow" down the optimal scaling, steepness and pre-ring settings as well as the type of noise shaping (just the dither - PSP X-Dither - has 27 different settings, but some were pretty easy to dismiss with even casual listening as they were far too aggressive, producing too much quantisation noise at the extreme top end).

 

[amirm]

Just to give people an example - I have been doing a project over the last 4 years of transcribing my vinyl collection to CD. My transcription masters are 24/96 so I had to chose the appropriate resampling and dithering products and settings for CD production. I spent about 12 months coming up with the chosen products and settings purely for the CD production part of the project. This involved around 200 x A/B listening sessions and 60 burned CDs. The A/B sessions were required to "narrow" down the optimal scaling, steepness and pre-ring settings as well as the type of noise shaping (just the dither - PSP X-Dither - has 27 different settings, but some were pretty easy to dismiss with even casual listening as they were far too aggressive, producing too much quantisation noise at the extreme top end).

Love to see a thread on this topic and read how people are going about digitizing their LPs.

 

[Robh3606]

Depends on what and why you are doing it. I use it when I do speaker comparisons of DIY projects against a production speaker. For that its easy just match gain settings and go. I find it fun to see how the DIY's do even it isn't so good. At least you can turn it into a useful learning experience.

Rob

 

[APP]

I am highly motivated to them but the moment I think about doing it, the hating starts. Few things are more boring than doing these kinds of tests so I only do it when I am really, really motivated. Then I go to do it and hating starts.

+1 but if you do it with a friend AND a couple of bottles of wine it can be fun.

 

[Fiddle Faddle]

Love to see a thread on this topic and read how people are going about digitizing their LPs.

Be careful what you wish for! It might actually encourage people and in all seriousness, I for one would actually dissuade people from doing it. there is so much to learn if you want absolute top-notch results (and I accept nothing less) that I could almost hold a 6-month long full-time course in it or write a pretty thick book about it. If I had my time again I wouldn't have started doing it to be honest - it is a HUGE undertaking even if you want a 95% transparent result to the original analogue source, let alone anything better.

OK, for me I might be able to start sitting back and relaxing now after 4 years and enjoying the fruits of my labour. And yes, my results are better than any commercial CD I have heard. But, they are not better than a 24 bit, 96 KHz download or Blu-ray disk of the same (assuming they all come from the same source which in my case they do for the most part).

It took me a year full-time experimenting and learning, refining equipment choice and refining technique as well as constructing the workstation and that was well before I even got to the stages I described in my previous post (actual CD production). Just my operational flowcharts and checklists that describe the full transcription process step by step on my setup run to about 20-plus pages. And to add insult to injury, a couple of years into the project 24 bit, 96 KHz downloads and even blu-ray audio disks became far more common - specifically targeting the type of material I am interested in - classical music recorded during the "golden age". And to my ears - as I mentioned in the previous paragraph - these high resolution sources will beat a vinyl transcription any time, regardless of how good the equipment used is and no matter how brilliant and golden-eared the "engineer" is (again, assuming the downloads or blu-ray disks come from the identical source material as the vinyl which is the case for me).

But of course I was already so heavily committed in terms of time, money and resources that I wasn't going to stop half-way through and change to buying the downloads - especially when I already have a decent vinyl collection. But what I have learned in very broad terms from my own experience is this:

You need highly-refined critical listening skills because it is not just a case of pressing buttons and dropping the tonearm. You need to know what thresholds (what, how, why, when) to push for correction of common vinyl arfefacts, you need to learn do it both manually (sample by sample) as well as automatically - but changing the settings each time because every record is unique. You need to listen to each and every single edit you make through both speakers and headphones, otherwise you run the risk of introduce phase issues, since these are an un-wanted (but avoidable) artefact of many correction algorithms.

You need a brilliant analogue to digital converter, otherwise there is no point. The losses between the analogue feed and the digital output from the converter are surprisingly obvious, even with expensive equipment. So if you are not up to borrowing or using something like a dCS converter (even a vintage one from the early 1990s will give excellent results), you are going to change the sound quite noticeably even before it hits the workstation. Then when it hits the workstation hardware and software you need to be a master of that too. Mine runs on Windows and even changing settings you'd think should make absolutely no difference whatsoever to the end-sound will, infact, change the sound. This is especially the case when the software runs on a complex operating system that was never designed from the ground-up to be used specifically for audio mastering. You can trim the OS to the bone and tweak it till the cows come home - it will still change the sound.

So the bottom line is that I grapple to see the point these days when so much material is being released at high resolution and from original master tapes. Just look at the massively expanding download catalogue at Acoustic Sounds, for example - it's all audiophile stuff from the same sources used to make the audiophile LPs.

As I say, if I had my time again I would have sold the entire vinyl collection and just bought downloads in lieu. What I have definitely found is that the two main reasons commercial CDs sound bad (when they really ought not to since the high resolution equivalents sound great) are these:

(I) the resampling algorithms and dithering algorithms used are not as good as the state of the art ones available today, so there is a substantial loss from the original "flat" 24 bit, 96 kHz transfer from the analogue tape to the final 16/44.1 CD master file. Yes, there is always going to be a loss - it is quite noticeable too. I spent a year full-time trying to make this particular processing step transparent and failed miserably. But it doesn't have to be nearly as bad as what we hear on commercial CDs. There are better products and better settings available today, but it takes time and extremely refined listening skills to find the right settings. You have to know what to look for and you have to know exactly what the instruments sound like in real life if you are to manage this part of the process properly.

(II) Yes, this is going to be incredibly controversial, but I don't care - different blank CD-Rs sound different, as do different burners. Again I spent a year and about $1,000 just going through different brands of blanks and burners. In the end I use an expensive blank from Japan and expensive Pioneer burners only available to the Japanese Domestic Market. Depending on the CD player you are using, this can also create a substantial difference in sound. The CD blanks and burners I use are about 98% transparent subjectively to the CD master file. But that is way better than a commercial CD which I would typically rate at around 85% to 90% for the same.

Of course, if one just wants high resolution copies then the CD stuff doesn't matter, but even then there are tricks to watch. It goes on and on but the point I guess I am making is that I simply would not recommend it in this day and age. It is highly involved, extremely time consuming and the results simply cannot - by any definition you choose - equal the quality of a modern high resolution download procured from exactly the same source.

Sorry about the diversion and getting OT. But I had to put it out there. You really need to be a bit of a lunatic to grapple with it and I don't mind admitting that I must be one!

 

[Rodney Gold]

If I can instantly and seamlessly AB 2 items Im considering , I will
However setting up level matched instant AB comparisons in a domestic environment is not possible for lots of items
Can't do it with speakers , very difficult with speaker cables and amps , interconnects , difficult with pre amps , ok for dacs , dsp devices etc..

the problem with digitising vinyl (needledrops) is that results are a snapshot of the cart , phono stage , arm , table at that point in time. Any time you change anything that changes the sonics the needledrop changes...so a case can be made to AB these to get a handle on the exact difference your vinyl gear change made

 

[Fitzcaraldo215]

Rodney - you have touched on one of my pet peeves, and it is one of the many reasons that vinyl and I have parted ways some time ago. It is darned near impossible to A-B two cartridges under identical conditions. And, cartridges are, I dare say, the main contributor to differences in vinyl's ultimate sound, though some may disagree. Dealers will not lend them, and there are seemingly few dealer turntable setups with identical arms which can be fed to the same phono stage and system. I will not get into all the other setup tweakeroni - optimal tracking forces, VTAs, cartridge loading, etc. I think you inevitably buy a phono cartridge with less information about how it is going to sound than is available about other system choices. As a result, the well heeled vinyl-philes I know usually have an assortment of cartridges pre-mounted in head shells, multi-arm TT's, etc.

Most everything else in my system can be decently A-B'ed focusing on just one component swap, not that it is a perfect means of comparison. But, for the average Joe Audiophile, what is better?

 

[amirm]

Be careful what you wish for! It might actually encourage people and in all seriousness, I for one would actually dissuade people from doing it.

Thanks a bunch for writing it. I am dissuaded already . I had a Jazz CD that did not have metadata. I sat there way, way too lazy to type all of that. Then I remembered my question to you and how I would have to sit there after ripping and add the metadata and thought, well, that ain't worth it .

Thanks again.

 

[GaryProtein]

VERY NICE!

I'm impressed with your 7.1 device!

I'm sure the six inches of wire in the box has a minimal if any effect on what comes out.

 

[audioguy]

I despise A/B testing, but it has saved me a lot of money. But because I am so subject to expectation bias, I only do so blind (speakers not included). I found this device that allows me to do blind testing as all of my equipment is in the room just behind my front wall. It has a button to switch between the two inputs and another individual pushes the button so I can not see what I am listening to. Is this box perfectly transparent? Probably not but then neither is anything else. And I have gotten to the point where if the differences between two components is masked by whatever lack of transparency is in this box, I wouldn't buy it anyway.



It works for anything that uses XLR cables (DAC's, preamps, amps, etc)

I discovered this little box when I was looking for a way to (blindly) compare two 7.1 SSP's so assembled a "Rube Goldberg" device consisting of 4 of them and a spring-loaded bar that would activate all of them at the same time. Worked like a champ. (This device is now for sale should anyone be interested. Cost of parts was just over $600). The unpainted wood part is the base and the black handle is the spring loaded push bar.

 

[Jinjuku]

I find the blinded A/B process educational. So for that aspect I like it. I just A/B'd a setup with the ~$40 NEO 3.5 waveguide tweeter with a much more expensive (5 times the cost) tweeter.

Didn't find out what was what till after. Now my preference was for the more expensive tweeter but man did it get a run for it's money. Others liked the less expensive tweeter.