Converter loop-back tests

[Keith_W]

I have no idea. I think the odds of getting four out of four correct are 2^4 or 1 in 16

No, it is 1/(4!) i.e. 4 * 3 * 2 * 1 = 1 in 24.

(edit) oops I see that someone else has pointed this out.

 

[rbbert]

Roy's moved within the last year. Now they're next to the golf course in Kaanapali. We're going on Saturday.

Merriman's in Kapalua is very worthwhile (especially the beignets), and Pacific O is a perennial favorite of ours as well.

 

[Ethan Winer]

If there is any legitimate interest in this test, a new thread should be started. Amir, given your concern over Ethan's involvement, and given your capabilities, perhaps you could start the thread? Maybe even perform the generation test with a few different tracks that would more readily as well as less readily (if at all) be detectable.

Since that hasn't been forthcoming, nor has Amir even committed to doing it eventually, I created a new set of files and wrote a corresponding article on my web site:

Converter Loop-Back Tests

.....moderator edit..... at least I'll get guesses from many other people over time. Anyone who was shy about posting their guesses publicly here is welcome to email me through that article. At some point I'll reveal the stats, but I never reveal people's names from my various other "guessing" tests.

--Ethan

 

[Bruce B]

Another forum member pointed me to this thread a couple days ago. Ethan, I have you on Ignore..... now I'm glad I didn't see it sooner.

 

[Keith_W]

I imagine the same people who made bold claims won't chime in there either,

A few days ago you were pulled up by the moderators for having a go at your forum enemies. Now you are at it again.

 

[RBFC]

A few days ago you were pulled up by the moderators for having a go at your forum enemies. Now you are at it again.

Everyone, please hold yourselves to a high standard. There are many "guilty" parties in this thread. The premise under discussion is very interesting. Watching the frequent volleys back and forth is very fatiguing.

Lee

 

[amirm]

Since that hasn't been forthcoming, nor has Amir even committed to doing it eventually...

I don't have much interest in this type of test Ethan. Generational loss is not a concern for consumers enjoying music. They don't make copies of their music to play (putting aside exception of digitizing LPs).

If you want to volunteer to do blind tests, how about running some comparing say, diffusion to blank wall to absorption for first reflection? You have the means to do all of that and it would create really good data. You and I have had this discussion before and I am still lost as to why you don't apply removal of bias to acoustic situations and spend all your energy here.

You always say the room matters most. Why not apply some blind testing energy to that?

 

[Ethan Winer]

Generational loss is not a concern for consumers enjoying music.

Consumers do care about audio quality, and using multiple generations is the best way to show the degradation of a converter by listening alone. Measuring will show the changes after only one pass, but most people don't have access to test gear, or the knowledge to use even freeware testing software. So in fact this test is very useful to anyone who cares about converter quality. If you can't hear the degradation after one pass, or five passes, then the converter is transparent enough that you can move on to more important things.

If you want to volunteer to do blind tests, how about running some comparing say, diffusion to blank wall to absorption for first reflection?

That's not a bad idea, but it's not as easy as you suggest. It would require literally a week of preparation, finding volunteers, setting up the room, upsetting my home studio and bringing the speakers and a CD player, and so forth. But the real reason I have little interest in such a test is because the difference between a bare wall, absorbers, and diffusers is so obvious there's no need for a blind test. Everyone can easily hear it and decide immediately which they prefer. Versus my converter tests where the differences are very subtle.

--Ethan

 

[Ethan Winer]

A few days ago you were pulled up by the moderators for having a go at your forum enemies. Now you are at it again.

I am not "having a go" at anyone. All I'm asking is for people who made bold claims to back up those claims. If that's seen by some as being rude, so be it.

--Ethan

 

[RBFC]

Consumers do care about audio quality, and using multiple generations is the best way to show the degradation of a converter by listening alone. Measuring will show the changes after only one pass, but most people don't have access to test gear, or the knowledge to use even freeware testing software. So in fact this test is very useful to anyone who cares about converter quality. If you can't hear the degradation after one pass, or five passes, then the converter is transparent enough that you can move on to more important things.



That's not a bad idea, but it's not as easy as you suggest. It would require literally a week of preparation, finding volunteers, setting up the room, upsetting my home studio and bringing the speakers and a CD player, and so forth. But the real reason I have little interest in such a test is because the difference between a bare wall, absorbers, and diffusers is so obvious there's no need for a blind test. Everyone can easily hear it and decide immediately which they prefer. Versus my converter tests where the differences are very subtle.

--Ethan

Ethan,

I would submit that it would indeed be worthwhile, as you would then have scientific data to act as the foundation for the case supporting room treatments. Specifically, graduated application of YOUR room treatments and their effect on the measured frequency response of the room. Yes, it would take some work, but a week of work to support your business seems like a reasonable contribution.

Lee

 

[amirm]

Consumers do care about audio quality, and using multiple generations is the best way to show the degradation of a converter by listening alone.

Why? You are simulating effects of no interest. Let's say the converter has 0.2 db of roll of above 18 Khz. By the tenth generation you are down 2 db. Someone with good hearing may be able detect that loss. Why would that related to anything we care about here?

Measuring will show the changes after only one pass, but most people don't have access to test gear, or the knowledge to use even freeware testing software. So in fact this test is very useful to anyone who cares about converter quality. If you can't hear the degradation after one pass, or five passes, then the converter is transparent enough that you can move on to more important things.

I ask again why? The argument is over analog vs. digital. Let's say I take a color picture and make a black and white version. Then make 10 more copies in black and white. Are you saying that if the 10th version of that black and white copy is the same as the first, it somehow indicates the color was never lost?

Mind you, this is not my argument. But it is for others. They think the very nature of digital causes something to be lost. If so, you can't present this test as being useful in that regard.

That's not a bad idea, but it's not as easy as you suggest. It would require literally a week of preparation, finding volunteers, setting up the room, upsetting my home studio and bringing the speakers and a CD player, and so forth.

Ah! You ask us a bunch of people here to burn CDs, run tests, etc. But if we ask you to do some work that is too much? Why? Is the search for audio knowledge not worth your time but should be worth ours?

But the real reason I have little interest in such a test is because the difference between a bare wall, absorbers, and diffusers is so obvious there's no need for a blind test. Everyone can easily hear it and decide immediately which they prefer. Versus my converter tests where the differences are very subtle.

Just because there are differences doesn't obviate the need for the test whatsoever. People are constantly asking which is better. Vast majority of blind testing done in the industry is along those lines. Only on forums where people want to trap each other is there such strong fixation on "is it or is it not there." Take audio compression where the most blind tests are applied to it. In that domain we know the sound is different yet we still try to quantify which is better. Bias plays a role there just the same. People have run blind tests with respect to speakers and found different results than sighted. http://seanolive.blogspot.com/2009/04/dishonesty-of-sighted-audio-product.html

Two out of four votes got inverted in blind testing. Can you please explain why bias is not a factor and of interest to rule out in your view?

 

[Ethan Winer]

I would submit that it would indeed be worthwhile, as you would then have scientific data to act as the foundation for the case supporting room treatments.

I already have mountains of scientific data that acts as a foundation for supporting room treatments! It's all over the RealTraps web site. There are Before / After graphs of modal ringing, frequency response, improved RT60, reduction of comb filtering with absorption at reflection points, and so forth. What would a blind listening test accomplish? We have many dozens of testimonials from people who hear the obvious improvement. Blind tests are needed only when differences are subtle, to weed out "I think I might hear something but I'm not sure." Room treatment is not like that. Converters are.

--Ethan

 

[Ethan Winer]

Why? You are simulating effects of no interest. Let's say the converter has 0.2 db of roll of above 18 Khz. By the tenth generation you are down 2 db. Someone with good hearing may be able detect that loss. Why would that related to anything we care about here?

I don't know how much clearer I can be. In other threads here people have claimed to be able to hear the degradation from even one pass through the finest converters money can buy. So this lets them prove it. Of course we both know they won't do that, and we both know why. So I've moved along and put this out for the public. It's already generated huge interest on my Facebook page, and a few minutes ago I posted a link in a professional recording forum where I'm sure people will chime in without fear of looking foolish.

You ask us a bunch of people here to burn CDs, run tests, etc. But if we ask you to do some work that is too much? Why? Is the search for audio knowledge not worth your time but should be worth ours?

Oh geez Amir, really! Most audio professionals and audiophiles already have a way to play Wave files through their good system. And if they don't, burning a CD takes five minutes. Further, I do this for the benefit of others, not myself. If they'd rather not know the truth about converter quality, or need to pretend their ears are superior to win clients, that's their problem.

Anyway, as I said I've moved along. Nothing more will be accomplished here unless people post their choices. Which we know they won't so I'm done. I revived this thread only to post the link to the new test files. BTW, I just added a third set of files to further reduce the chance of random correct guesses.

--Ethan

 

[RBFC]

I already have mountains of scientific data that acts as a foundation for supporting room treatments! It's all over the RealTraps web site. There are Before / After graphs of modal ringing, frequency response, improved RT60, reduction of comb filtering with absorption at reflection points, and so forth. What would a blind listening test accomplish? We have many dozens of testimonials from people who hear the obvious improvement. Blind tests are needed only when differences are subtle, to weed out "I think I might hear something but I'm not sure." Room treatment is not like that. Converters are.

--Ethan

On one hand, in this thread, you call out individuals to be documented as to their ability to hear differences. While these differences are often measurable, the question about their audibility then arises. So, while one can present "mountains" of measurements, I still maintain that it would be useful for you to have documented blind tests that prove that a population of listeners can correctly identify the use of room treatments. Otherwise, the assertions put forward are really no different than those stating analog/digital discrepancies. Both measurements AND statistical proof of audibility in controlled testing should be provided.

Lee

 

[mep]

We have many dozens of testimonials from people who hear the obvious improvement.
--Ethan

So does Shakti Stones. http://www.shakti-innovations.com/reviews.htm

 

[amirm]

I don't know how much clearer I can be.

Oh you are clear. Just not convincing .

In other threads here people have claimed to be able to hear the degradation from even one pass through the finest converters money can buy.

That one transform takes you from analog to digital domain. Did you read the analogy I mentioned? I constantly run into this problem with folks at the extreme point of view of objectivity. They use their own conclusions in creating tests to convince the other camp. I don't understand why you don't see the circular nature of this.

If you are unbiased, and someone says there can be a degradation when going from analog to digital -- any kind of digital -- you have to assume in construction of your test that such a thing can be true. If so, then taking already digital content and testing it further doesn't test that hypothesis. It tests something else no one cares about which is generational loss.

So this lets them prove it. Of course we both know they won't do that, and we both know why. So I've moved along and put this out for the public. It's already generated huge interest on my Facebook page, and a few minutes ago I posted a link in a professional recording forum where I'm sure people will chime in without fear of looking foolish.

Great. Finally you are following my advice to go and run the test elsewhere . As to fear of looking foolish, I am going to book that you are afraid of running blind tests against acoustic products and none. Hope you are happy with that just the same .

Oh geez Amir, really! Most audio professionals and audiophiles already have a way to play Wave files through their good system. And if they don't, burning a CD takes five minutes. Further, I do this for the benefit of others, not myself. If they'd rather not know the truth about converter quality, or need to pretend their ears are superior to win clients, that's their problem.

Vast majority of people use portable devices to access the Internet. Most of these have no CD drive. I know I sure was lazy enough to do anything on my laptop than to run a simple analysis. See below . I am also really, really surprised that you still have not figured out which ways totally invalid results can be tested.

Anyway, as I said I've moved along. Nothing more will be accomplished here unless people post their choices. Which we know they won't so I'm done. I revived this thread only to post the link to the new test files. BTW, I just added a third set of files to further reduce the chance of random correct guesses.

--Ethan

Well, we could have accomplished a lot but you didn't want to go there. You are just interested in winning a game with no knowledge of audio discussed in the middle. That is disappointing because people see you coming from a mile away and game is not played. What could have therefore been accomplished is a) learning the limitations of such online tests and b) science of audio/DAC/ADC. None of this seems to interest you. Sad because if we had such a discussion, we could have all learned something.

Let me give you a taste of the type of discussion we could have had. Here are the answers and statistics generated with regards to RMS power as reported by Adobe Audition:

File name: Min/Max RMS Power: Difference
Original: focusrite_b.wav: -56 db/-7 db: 49 db
1 generation: focusrite_d.wav: -92 db/-7 db: 85 db
5 generations: focusrite_a.wav: -87 db/-7 db: 80 db
10 generations: focusrite_c.wav: -83 db/-7 db: 77 db

You and Arny have been strong champions of audio not having enough resolution to even merit 16 bits. When I ask why, such reports are immediately put forward. Isn't it interesting that the stats are all over the place here? The so called dynamic range is varying from 49 dB (for the original file no less) to 85 dB. I asked you earlier if you had messed with the copies. You said no. If so, how do you explain this? How did the noise floor drop to -92 db from -56 dB?

To be clear, in my view these stats are completely bogus. But as I said, you seem to live by them so here they are .

 

[Ron Party]

It tests something else no one cares about which is generational loss.

Really? I could swear both Lee and I posted we were. I also would suggest that anyone who bothered to take the time to take the test were interested. I also would suggest that many others are interested but could not get past their personal grudges and, as such, I suggested you run the test.

Well, we could have accomplished a lot but you didn't want to go there. You are just interested in winning a game with no knowledge of audio discussed in the middle. That is disappointing because people see you coming from a mile away and game is not played. What could have therefore been accomplished is a) learning the limitations of such online tests and b) science of audio/DAC/ADC. None of this seems to interest you. Sad because if we had such a discussion, we could have all learned something.

I, for one, would welcome any such discussion. I'm sure I could have learned something but, alas, it is apparent that personal grudges prohibit the same. I am saddened that we can't elevate ourselves to that level of discourse.

 

[mep]

Ron-Your dismay might just be misguided if Ethan really did cook these files as Amir’s data clearly shows. There is no way that successive generations of conversions are going to increase dynamic range in a recording.

 

[Ron Party]

Hi Mark. Yes! I wish to extend to you an honest thank you. I am interested in the substance, not personal grudges. I have no doubt Steve, as Site Founder, is interested in exactly the same.

If the test is faulty, beautiful. Let's examine why. If the test is not faulty, also beautiful. Let's examine why. In either case, we learn! Fantastic!

In either case the discussion is elevated to a level where I sincerely hope we can comfortably reside. No longer are we focused on the poster. Instead we proceed as if everyone posting to this (and, really, any) thread had no user ID but instead was anonymous.

 

[amirm]

Really? I could swear both Lee and I posted we were. I also would suggest that anyone who bothered to take the time to take the test were interested.

I took the test but was not interested . Why are you interested Ron?

I also would suggest that many others are interested but could not get past their personal grudges and, as such, I suggested you run the test.

Curiosity, sure. But interest? In what way an audiophile runs into generational loss in digital audio? What audio purchase decision changes because of it?

Strawman. Your use of the smiley does not make it any less so.

Strawman? It was simply tit for tat. He says if someone doesn't take the test they are afraid of the outcome. I say since he has all the means *and motivation* for blind test yet he does not conduct any in the field of acoustics, he is worse off than others here. Let me ask you this. You say you are interested in generational audio tests. What is your interest level relative to that when it comes to acoustic products? Less or more? Do you think Ethan can help us more or less with that question than this one?

I, for one, would welcome any such discussion. I'm sure I could have learned something but, alas, it is apparent that personal grudges prohibit the same. I am saddened that we can't elevate ourselves to that level of discourse.

On whose behalf Ron? Mine? Remember, I took the test. You did not by the way . I have no fear of any discussion. But I am really, really tired of the two camps fighting for the sake of fighting. I can tell that is all it is when no technical information is shared. And that is what we have here.

In the last few months I have been deeply studying listening tests of acoustic science. I have tried to discuss the same elsewhere. All the people who fight for listening tests in every chance they get, have been fighting me tooth and nail as to relevance of listening tests there. They much rather go by gut feeling and measurements all of a sudden than performing listening tests. So my request of Ethan is not a random one especially when he was present in the other discussion. I have asked him the same before and as here, he puts forward that incorrect answer that just because there are differences, blind tests are not necessary. That makes no sense to me. Either you believe there is a need for such removal of bias or there isn't. Ethan says the room is more important than anything else. How many blind tests of room acoustic products do you think I can find in his book, forum discussions or other?

Back to the what I have been doing, I can tell you that what I have learned is 1000 times more interesting and relevant to sound in our room than this test. You want to completely change your outlook on audio, study what I have been doing. I wrote an article for WSR magazine in the last issue which I will be posting soon on that topic relevant to true dynamic range of our room using that science. I just finished another one for 20th anniversary of WSR magazine that goes into far more depth on how psychoacoustics and listening tests turn upside down much of what we think we know about acoustics in our rooms. This is why I didn't jump and run around to find a test clip for this test. There are much more useful things we could spend our time on than to run a public test to poke the finger in people's eyes who have gotten a test wrong. I for one am not in favor of public hanging of the forum members in that regard. I rather we all work collectively to learn something new. In the words of the Chuckie (Ben Affleck) in my favorite movie Good Will Hunting: I may not know much, but I know this."