Page 6 of 18 FirstFirst 12345678910111213141516 ... LastLast
Results 51 to 60 of 175

Thread: Audible Jitter/amirm vs Ethan Winer

  1. #51
    Banned
    Join Date
    Apr 2010
    Location
    Seattle, WA
    Posts
    16,046
    Quote Originally Posted by Phelonious Ponk View Post
    Double blind simply means that neither the subject, nor the people conducting the test know which sample is being tested when. You may have accomplished the same thing through your informal methodology (I've used exactly the same methodology myself, by the way), but you were both conducting the test and the subject of the test. The X is the core of the test. Randomization is typically applied through software in this kind of testing where you'l be presented with A, presented with B, then presented with X, which could be A or B, and asked to identify it.
    Exactly. I was trying to get rsbeck to answer this way because then he would realize that what he is asking for and what I ran are extremely close to each other. To expand, the distinction for double blind is to remove conductor bias. A conductor for example, could on purpose or not, change how the test is conducted. For example, he could give me one sequence of clips, and you a different set after hearing whether I am an audiophile or not. That could impact the outcome in a positive or negative way.

    If I had my wife do the switching with my back turned, we would all agree then that this would fall in the classic category of double blind because of the presence of a person conducting the test without knowledge or ability to change how the test was run. But think about it. It would not add any more accuracy to me randomizing the sequence, and then switching them without knowing which one is which.

    The distinction remaining for the readers is then whether I did the randomization correctly or not. If I did, no more value comes from having someone else conduct the test versus me.

    Your methodology may have been "blind enough," but it wasn't AB/X. By definition it can't be if you were doing this alone and going back and forth between two examples instead of randomly presenting A, then B, then X, identifying EX, and repeating.
    ABX is an example of double-blind methodology but not the the definition thereof. It is designed to remove the chances of cooked sequences and the test harness itself introducing bias in some manner. We also use it to find out which testers were just guessing and had no idea which one was which. This aspect however, does not apply to trained listeners. There are also some other distinctions I won't drill into here.

    I personally put a lot of value in people running a blind AB test and I am confident Ethan does too as those are the type of tests he has put forth in his talks and on his web site. Sure, some more confidence comes from having ABX style testing especially if you don't have the knowledge of how a test was run and who ran it. But substantial amount of validity comes from the fact that the experiment is blind.

    And you may have done enough trials to reach a statistically significant sample, but if you did, you have a lot more patience than I. I've done the "push the button until you forget where you are, A/B for awhile, then look" test you've described. I want to declare it audible or inaudible pretty quickly then get on to something more productive or entertaining.

    P
    I actually take it to one more level. I change which source was which, re-randomize and run the test again. At the end, I look to see which one was which. If the results were consistent, then I would believe them.

    Above is important to do due to variations in hardware and content. For example, it might seem that if I have two identical CD players and two of the same titles, then switching sources should not be necessary. But jitter and similar audio artifacts can occur due to very small differences and we need to rule them out. Tolerances in say, power supply components could let out more AC ripple on to the DAC clock source in one player or the other. Or one CD could have been stamped differently than the other.

    Fortunately I have never found switching to disagree with the first set of results but I don't sleep easy until I can verify the difference a second time.

    Running a test twice also rules out other things that could make it wrong such as the starting sequence not being random.

  2. #52
    Site Founder And Administrator Steve Williams's Avatar
    Join Date
    Mar 2010
    Location
    Coto De Caza, California on the 13th fairway of the south golf course
    Posts
    25,956
    I remember this discussion in another forum last year where you were taken to task on this but you maintained your position admirably. Although not strict ABX I am hoping people understand your methods
    Steve Williams
    aka oneobgyn
    There's ALWAYS another Steve Williams BUT there's only "oneobgyn"
    International Distributor of Center Stage Feet and owner of PitchPerfect Sound (www.pitchperfectsound.com)
    Dealer Lamm Electronics
    My System

  3. #53
    [WBF Founding Member] Moderator RBFC's Avatar
    Join Date
    Apr 2010
    Location
    Albuquerque, NM
    Posts
    5,057
    After reading through the description, I'd say that Amir's methodology would most likely replicate the results and procedural requirements to function as a double-blind test. However, as stated, it could not be reported as an "official" double-blind test due to the variances in the conduct of the test.

    Personally, I feel it's important as audiophiles that we learn how to assess equipment via methods like this. Since we often have only ourselves to perform the work, developing standardized one-operator protocols for "blind" listening seems like a prudent idea.

    Lee

  4. #54
    Banned
    Join Date
    Apr 2010
    Location
    Seattle, WA
    Posts
    16,046
    Quote Originally Posted by rsbeck View Post
    Would also beg the question why (sort of) single blind rather than the more credible double blind or ABX. I think Ethan has already stated that all you need is one person to pass a DBT or ABX to back your claim and he would change his opinion.
    I don't think he has asked for ABX. As I noted, he uses AB tests to prove his own points so it would be odd to all of a sudden require a high level from the other person.

    As for wanting that one person, he has that: me! Have him cross examine me all he wants to get comfort in that I knew what I was doing. Only then does he get to dismiss my results.

    On my side, I have gotten him to say that he has heard the very distortions he says don't exist when going from 16 to 24 bits. His defense is that he had to turn up the volume to hear them. I am fine with that. As long as he agrees that the distortion is indeed there, and can be heard at elevated levels, then we are a hell of a lot away from "you can't hear jitter no matter what." If you want to turn this in a double blind test, I could force him to use the very clip he used in that test, turn up the volume as he did, and I would win! After all, we know that he could hear the difference.

    Now if you want to tell me that he likes to cook the tests by using loud material so that I can't turn up the volume as he has done in his test, then we don't need to do that either. I have already conceded that there are a million ways jitter is not audible and that digital audio fidelity is not about loud signals. What he has to prove is that jitter is never audible. If he indeed heard something when he turned up the volume, then what better evidence do we need?

    Oh yes, proving that what he heard was due to jitter. I don't actually know if it was since I can't measure his gear. But I would declare victory anyway since it doesn't matter why the fidelity was not there. His own test proves that accuracy does matter and that digital is not perfect in all of its forms. That is something he would argue against just as much as he jitter.

    Indeed, in this discussion we are trying to figure out if jitter reduces effective resolution below 16 bits, does it matter? If he has found that there is a difference between 16 bits and 24, surely he has conquered hearing a much more difficult impairment than the one we are talking about here!

    It is a subject for another test but I suspect the test he really ran above was 16 bits versus lower than 16 bits. Most "24-bit" converters are not linear above 16. Run them in 16-bit mode, and they wind up only being good to 14 bits. So I shouldn't be unfair to him and make the last point above . BTW, he would only know this if he had seen measurements of his DAC linearity as I have for all the devices I have tested. I am always disappointed when the advocates of objective testing, do away with measurements because it is too hard, yet they demand others to go through a lot work to prove their point of view. Either you believe in objectivity or not. Start with measurements and then move on to listening tests which tend to be subjective at some level.

  5. #55
    [WBF Founding Member] RUR's Avatar
    Join Date
    Apr 2010
    Location
    SoCal
    Posts
    647
    A partner can make life much, much simpler... When I evaluated several DAC's last year, my partner used a die to choose which DAC was to be connected for any given trial. He'd roll the die, connect the indicated DAC (or no change, if it was the same DAC as last trial) and leave the room. I'd walk in, listen for a minute or two and select which DAC I thought was in-use. My view of the gear was completely blocked i.e. no visible clues. Volumes were level-matched to within 0.1dB. The big drawback to this technique is acoustic memory - each observation was separated by approx. 2 minutes, so the differences need to be significant enough to be heard after such a long time lapse.

  6. #56
    Now if you want to tell me that he likes to cook the tests by using loud material so that I can't turn up the volume as he has done in his test, then we don't need to do that either.
    I would argue that riding the volume up during quiet passages, or any manner of playing the material at abnormally high volume for the purpose of pushing the distortions into audibility is cooking the tests. But with that said, Amir, your methodology, while not "double" and containing no "X" is sufficiently blind in my book. I've used very similar methods often, not to prove anything to anyone else, but to determine whether or not a new component is earning its keep in my systems. It works for me. FWIW, I use the high end of normal listening volume (too high for an extended listening session), on headphones. I call what I can't hear under those conditions "inaudible." But your hearing, your comfort with extreme volume, and your mileage may vary.

    P

  7. #57
    [WBF Founding Member]
    Join Date
    Apr 2010
    Posts
    848
    We may wish that we can be both the subject and conductor of the test and still call it double blind, but as long as you are both the subject and conductor, it isn't double blind. So, it is inaccurate to assert that you have conducted your tests double blind. Since your methodology is a sort of informal one where you claim to confuse yourself and then a/b, you still have all kinds of possibility for experimenter bias that cannot be ruled out. That's why double blind tests are conducted with another level of blindness. Scientists understand that even the most objective among them is subject to experimenter bias -- because they are human. IMO, even calling your test "single blind" would be problematic because you claim to be blinding yourself through intentional confusion and you cannot rule out experimenter bias. So, it isn't an insult to ask that experimenter bias be completely ruled out before calling your tests blind or double blind and it is hardly an insult to ask that correct terminology be used when reporting. Call it what it is -- it is a sort of single blind test. Calling for accuracy is not an attack nor IMO should anyone feel attacked.

  8. #58
    Yep, all kinds of opportunity for bias and inaccuracy, which is why I use that methodology for testing equipment in my own system, not to show evidence of anything to anyone else. I would still vote for the "confuse myself single blind A/B" comparison over any variety of sighted listening tests, even panels. Audiophiles who think they are giving themselves and their gear a fair shot with sighted listening "testing" are fooling themselves. Literally and figuratively.

    P

  9. #59
    Site Founder And Administrator Steve Williams's Avatar
    Join Date
    Mar 2010
    Location
    Coto De Caza, California on the 13th fairway of the south golf course
    Posts
    25,956
    I would still vote for the "confuse myself single blind A/B" comparison over any variety of sighted listening tests, even panels.
    That is too funny. I think you should "trademark" that term in audio at the very least
    Steve Williams
    aka oneobgyn
    There's ALWAYS another Steve Williams BUT there's only "oneobgyn"
    International Distributor of Center Stage Feet and owner of PitchPerfect Sound (www.pitchperfectsound.com)
    Dealer Lamm Electronics
    My System

  10. #60
    Banned
    Join Date
    Apr 2010
    Location
    Seattle, WA
    Posts
    16,046
    Quote Originally Posted by rsbeck View Post
    We may wish that we can be both the subject and conductor of the test and still call it double blind, but as long as you are both the subject and conductor, it isn't double blind. So, it is inaccurate to assert that you have conducted your tests double blind. Since your methodology is a sort of informal one where you claim to confuse yourself and then a/b, you still have all kinds of possibility for experimenter bias that cannot be ruled out. That's why double blind tests are conducted with another level of blindness.
    You can rule out experimenter bias by asking me questions. This is impossible to do this normally when a paper is presented and we can't question the participant. Hence the reason the protocols tend to be more rigorous. Such is not the case here.

    Keep in mind that even if a test is run double-blind, it can just as well be biased. Note how the paper being discussed started with less important type of jitter. No amount of test methodology saves you after you make that mistake. What is important then is to understand the true nature of what is being presented, rather than being impressed or disappointed with superficial aspects of it.

    Scientists understand that even the most objective among them is subject to experimenter bias -- because they are human.
    I hope your implication is not that I don't understand this lest you want me to catch you explaining to a Chinese chef how to make stir-fry .
    So, it isn't an insult to ask that experimenter bias be completely ruled out before calling your tests blind or double blind and it is hardly an insult to ask that correct terminology be used when reporting. Call it what it is -- it is a sort of single blind test. Calling for accuracy is not an attack nor IMO should anyone feel attacked.
    I don't need to tell you the difference between asking a question which is respectful of the person's experience and credentials, and the type of indirect questioning you do when you don't believe a person .

    You don't see me grilling Ethan as to whether he knew how to run his audio workstation software to inject noise properly at the levels he says he did. That's because even though there could be opportunity for mistake, and operator bias, I consider his qualifications and ethics high enough to have it be immaterial. More importantly, if I did mention it, he would likely take it as an insult that he doesn't know how to use the very software he makes a living from.

    90% of the information in my posts had nothing to do with my own testing anyway. What I wish is that you consider that data which is free of bias and comment on it, rather then spending the entire Saturday correcting me on whether I understand what single or double blind testing in audio circles means . You need to trust me that I know the difference and that I was after objective data to steer software development for a $14B business. You don't set yourself up to fool yourself when you have something like that at stake....

    So let's move on and talk about other points. I think we have beat this one to death.

Page 6 of 18 FirstFirst 12345678910111213141516 ... LastLast

Similar Threads

  1. Hi from Ethan Winer
    By Ethan Winer in forum Introduce Yourself
    Replies: 8
    Last Post: 07-12-2010, 10:14 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •