Conclusive "Proof" that higher resolution audio sounds different

[jkeny]

ArnyK, I'm sure you are correct as you should know your own files, by this stage. It's probably my confusion between Scott's files & your files - to be honest there were so many versions of these files released & with people just calling them AVS files it was easy to get confused.
So what you are confirming with your files is a time shift but no level shift (in the audio band)?

 

[jkeny]

Yes, Tim, best leave Amir to reply - I don't have the time nor inclination to read back through this thread.

 

[Orb]

I just reread all 90 pages of the thread and in fact find no such information. Furthermore I found the exact opposite being stated by a poster named Esldude.

I found a goodly number of other errors of fact, so some people around here received false benefits from my absence.

Please provide the post number in which this alleged discrepancy was found.

Not entirely true Arny as Esldude is open to possibility of it being dither/SRC related and this would be in the audioband.
That is my thoughts as well, and fits with Amir's context that ideally one should want the native source and also at DAC without it being "handled" in some way by the Studio (ADC to master file and also what is published) or DAC or software/etc.

Cheers
Orb

 

[arnyk]

ArnyK, I'm sure you are correct as you should know your own files, by this stage. It's probably my confusion between Scott's files & your files - to be honest there were so many versions of these files released & with people just calling them AVS files it was easy to get confused.
So what you are confirming with your files is a time shift but no level shift (in the audio band)?

Yup, no level shift but a trivial time shift.

 

[arnyk]

Not entirely true Arny as Esldude is open to possibility of it being dither/SRC related and this would be in the audioband.

Are you saying that his statement that there was no level difference can't be taken at face value?

As an aside the idea that dithering that is > 90 dB down would audibly affect levels doesn't pass any mathematical sniff test.

Looks to me like a lot of baseless speculation.

That is my thoughts as well, and fits with Amir's context that ideally one should want the native source and also at DAC without it being "handled" in some way by the Studio (ADC to master file and also what is published) or DAC or software/etc.

That pretty well describes the production of the "keys jangling" files. The output of the ADC driven by the output of the mic preamp was recorded no SRC or equalization into the actgual file that was circulated as part of the file set for ABXing. The resampled version differs only in that it was resampled twice as previously described. There was some editing and addition of other test tones, etc., but that did not cause any changes to the actual files for formal comparison.

There seems to be a lot of speculation about these tests.


Let us be clear: The most uncontrolled part of the test was the monitoring systems and actual procedures that the listeners used.

The only monitoring system that actually underwent any formal technical tests was AFAIK John Atkinson's. Some of his high end monitoring gear was pretty bad and had artifacts in the -50-60 dB range. Other of his high end monitoring gear was pretty good and the artifacts that were more like -80-90 dB down. However, we have no reports about any listening tests that were actually being conducted with it. I don't expect ever in my life to see some people publicly admit that they did even one of the dreaded ABX tests - it would be against their religion. Their Golden Ear cards would be permanently pulled by Harley and Co. ;-)

Other than JA's good work whose value is greatly diminished due to the lack of listening test results based on it, I know of no formal test results (i.e. results analyzed with test equipment) for any other monitoring equipment that was used. This is despite repeated requests to people who have alleged or demonstrated that they have the tools at hand to do them and made bold claims about the results of their listening tests.

So there is a lot of transparency about many important details of listening tests with alleged positive outcomes that seems to have gone missing. That of course detracts from their credibility.

Hence my comment that I would like to see what would happen if someone like JJ who lives in the general area, would proctor some tests.

 

[Phelonious Ponk]

Yes, Tim, best leave Amir to reply - I don't have the time nor inclination to read back through this thread.

We agree on that. I'm anxious to hear what Amir has to say, because if it is what you have paraphrased, it is in direct contradiction with what I've quoted from page 1. And Amir has never struck me as either careless or contradictory.

Tim

 

[esldude]

Recently some considerable back and forth has been due to the confusion about the AVS test files using AIX music involving I think 3 or 4 different songs and numerous threads about Arny's jangling keys file.

Post #1 in this thread was Amir saying he could ABX the jangling keys files.

Post #2 was about the AIX music files. Which subsequently have been found to have a level difference and what looks to be a sub-sample timing shift.

Speaking for myself Arny's files have a one sample timing shift which is trivial to correct. And which the FFT's below have had corrected before differencing. Though I thought there might be other artifacts initially, it is clear there is no level shift nor does there appear to be any sub-sample timing shift.

However, using Amir's methodology I was able to ABX 17 of 20 with Arny's original jangling keys and his conversion to 44/16 and back. I did this using the 2-4.9 second section of the file. I could hear no difference to point to, but had a feeling there was a difference. The feeling was apparently accurate it would appear.

I resampled the original jangling keys and could no longer detect a difference. It is my opinion whatever tiny, tiny difference I was hearing was due to dither or SRC artifacts (my guess is dither to 16 bit).

Attached are the FFT of differencing triangular dither, shaped dither, and Arny's original resampling against the original file. I could hear Arny's and not the other two.

I could take the difference file and filter out everything above 20 khz. Then amplify the difference. With Arny's resample I can hear jangling in the upper couple of octaves with only 16 db of amplification. With the other resamples I did you need 50 db of amplification and only just hear low level hiss or noise.

 

[arnyk]

However, using Amir's methodology I was able to ABX 17 of 20 with Arny's original jangling keys and his conversion to 44/16 and back. I did this using the 2-4.9 second section of the file. I could hear no difference to point to, but had a feeling there was a difference. The feeling was apparently accurate it would appear.

Exactly what is this Amir's methodology, exactly? Please do tell!

 

[sasully]

All disagreements are welcome Tony. We simply draw the line when it gets personal rather than discussing the technical topic.

If you noticed, I put proof in quotation mark for this thread. The "standard" of proof in countless forum arguments has been an ABX test. And such was suggested, and specifically foobar2000 ABX as the method of evaluating these files. It was thought until now that such ABX tests could not generate positive results. My testing and later that of others showed that positive outcomes are possible.

That a rather bold and self-inflating misrepresentation, and it's the sort of thing that keeps getting you a 'rep'.

What was 'thought' was that a rigorous double-blind comparison that is truly between 'hi rez' vs 'Redbook' formats will yield a negative result...except under extraordinary conditions (like, raising the volume so high that differences in background become audible in conversion that doesn't use dither and noise shaping. Which btw is exactly the sort of difference that Meyer and Moran reported). OF course too, if the comparator tool itself introduces a 'tell', the result could be 'difference'. Ditto if there is distortion at output. You aren't truly comparing 'hi rez to Redbook' in those cases.

So clearly no one says 'could not', point blank.

What they require is critical listening ability and motivation that these differences can be found. Negative outcome is assured if one thinks this is an impossible test. And not paying attention to smallest details, which in your case was the pop/glitch. All of this actually breaks new ground in these forum discussions. It shows that there are critical listeners which do much better than others in these tests. Even after the information you talk about was disclosed by others, vast majority of people still can't tell these files apart. So we have pretty strong proof that real "golden ears" exist.

Why is that important? Because expert listeners are a requirement for any listening test that is searching for small differences. But up to now, it was assumed that there is no such class of listeners. Therefore we could take the outcome of any blind tests, regardless of the skill of listeners, and apply it to the audiophile population at large. I don't remember reading about any of the DIY ABX tests that had expert listeners. Or training.

This is the most important lesson here. It should lead to more careful tests and more reliable results.

What *you* remember or not really doesn't matter.

For a *fact*, highly trained listeners have reported ABX results for comparisons on Hydrogenaudio. What comes immediately to mind are credible reports of hearing difference between 320 CBR mp3 and source, using good codecs. That's credible because people who tune lossy codecs -- and are thus 'trained' to hear subtle artifacts of mp3 encoding -- hang out there.

The *important lesson* here is not that training can increase discriminatory ability -- no one in their right mind disputes that -- it's that reports of difference need to be accompanied by consideration of the *possible sources of that difference*. Saying 'some people have better hearing than others' is only part of the story, and not the most interesting here.

No one here, you included, I am quite sure, actually hears the >20kHz content of the ' high rez' files. *Possibly* the bitdepth difference is audible, depending on how the conversion was done, and how abnormal the listening was (the fact that two variables -- SR and bit depth -- were changed is one of the issues making these tests problematic). Beyond that, we are left with either artifacts (from conversion, from the software , from the hardware) introduced into the audible band...or phenomena unknown to science-- and extraordinary claim require far more extraordinary evidence that provided thus far.

 

[esldude]

Exactly what is this Amir's methodology, exactly? Please do tell!

Well to listen and see if you find areas that sound slightly different to you. To narrow down and listen intensively to short segments. If it involved an entire song of a few minutes you don't listen to whole songs. You find a segment of a handful of seconds and compare those. And your files were a good example. Even at only 12 seconds I couldn't ABX them whole. But sliding around listening to parts here or there the 2.9 second segment I settled on was discernibly different.

 

[esldude]

Also to add a bit more on Arny's files. I mentioned filtering out the difference files above 20 khz and amplifying the result to hear jangling keys still. Going further I could filter out the difference file removing everything above 3 khz. That let me amplify it about 50 db and clearly hear jangling keys. Only when content above 1 khz was removed did I hear only noise when it was amplified.

So no reason to take it personally Arny. I would not have thought the tiny residual between those would be audible. And it almost is not. Usually is not. Only in the most picky circumstances is it barely perceptible. In general recreational music listening I can't see it mattering.

 

[Whatmore]

Let us be clear: The most uncontrolled part of the test was the monitoring systems and actual procedures that the listeners used.

This would also be true in the cases where no difference was heard-thus invalidating the null results. In which case, what did you hope to achieve by putting the files up for testing

 

[amirm]

That a rather bold and self-inflating misrepresentation, and it's the sort of thing that keeps getting you a 'rep'.

If you don't mind, on WBF forum we show more respect to each other than this. Please make your technical points and let that speak for itself rather than getting personal this way.

What was 'thought' was that a rigorous double-blind comparison that is truly between 'hi rez' vs 'Redbook' formats will yield a negative result...except under extraordinary conditions (like, raising the volume so high that differences in background become audible in conversion that doesn't use dither and noise shaping. Which btw is exactly the sort of difference that Meyer and Moran reported). OF course too, if the comparator tool itself introduces a 'tell', the result could be 'difference'. Ditto if there is distortion at output. You aren't truly comparing 'hi rez to Redbook' in those cases.

You bolded my sentence but then ignored it as you wrote this reply: "It was thought until now that such ABX tests could not generate positive results. "

"Such tests" means tests created such as ones by Arny where he said I think for some 14 years no one could tell the difference. Here is the intro to the test Arny created which I quoted in the first post in this thread:

Yes. Take the best audio system you can find. Take the best recordings you can find - recordings that sound great and also have significant content > 20 KHz, even > 35 KHz. Switch a 16 KHz brick wall filter in and out of the signal path. Nobody notices nuttin'

See? It says even a resample to 32 Khz is transparent. No qualifications is made as you are putting there. If you go to the link I provided to that conversation, this is what you see:

I did what I said in the previous post. I made recordings of live musicans in an exceedingly quiet and non-reverberent room @24/96 using 1/4" measurement mics that had strong content > 20 KHz, even 30 KHz.

I set up an ABX between the 24/96 files with a 16 KHz brickwall filter, and with full bandpass. I used speakers and amps with strong response > 30 KHz and put the listeners on axis of their supertweeters.

[Amir: where tests sighted or blind]
both

[Amir: And was the person performing the tests had his hearing tested to make sure they could hear above 16 Khz?]
Yes.
The training sequence was files brick walled at lower frequencies such as 9 KHz, and working up in logical steps.


[Amir: And was the person performing the tests had his hearing tested to make sure they could hear above 16 Khz?]
Yes.

So I ran this impossible test where folks that had hearing above 16 Khz could not perceive the difference even down to 32 Khz sample as Arny presented it. He has dithered the file. And I and others managed to hear the difference.

So clearly no one says 'could not', point blank.

Arny has clearly above. And I can find countless others who have said the same.

And here is you in a high-res to CD comparison on AVS: http://www.avsforum.com/forum/91-audio-theory-setup-chat/797512-196-24-vs-44-16-a.html#post9646092

If the player isn't doing any digital processing, 16 bits should be more than suifficient for any home use -- even though it won't be quite 16 bit in practice. IIRC, all other things being equal, people don't tend to hear bit depth reduction until it hits 14 bits (I think the pcabx site has a self-test for this). At home, the ~85-90 dB of dynamic range that '16 bit' CD provides should be more than enough. Most modern pop recordings and remasters don't even begin to take advantage of that range anyway.

There is no talk of dither, noise shaping, etc. What there is are empty claims of 14 bits being good enough. Where is your result of double blind tests that demonstrate that?

What *you* remember or not really doesn't matter.

For a *fact*, highly trained listeners have reported ABX results for comparisons on Hydrogenaudio. What comes immediately to mind are credible reports of hearing difference between 320 CBR mp3 and source, using good codecs. That's credible because people who tune lossy codecs -- and are thus 'trained' to hear subtle artifacts of mp3 encoding -- hang out there.

They also hang out here:

Not quite the topic of the thread but nevertheless part of similar arguments, here is a comparison of Arny's 24/96 file downsampled to 16/44.1 compared to 320 kbps MP3 of the same:

=================

foo_abx 1.3.4 report
foobar2000 v1.3.2
2014/07/19 19:45:33

File A: C:\Users\Amir\Music\Arnys Filter Test\keys jangling 16 44.wav
File B: C:\Users\Amir\Music\Arnys Filter Test\keys jangling 16 44_01.mp3

19:45:33 : Test started.
19:46:21 : 01/01 50.0%
19:46:35 : 02/02 25.0%
19:46:49 : 02/03 50.0% << dog barked in my ear wanting to go out
19:47:03 : 03/04 31.3%
19:47:13 : 04/05 18.8%
19:47:27 : 05/06 10.9%
19:47:38 : 06/07 6.3%
19:47:46 : 07/08 3.5%
19:48:01 : 08/09 2.0%
19:48:19 : 09/10 1.1%
19:48:31 : 10/11 0.6%
19:48:45 : 11/12 0.3%
19:48:58 : 12/13 0.2%
19:49:11 : 13/14 0.1%
19:49:28 : 14/15 0.0%
19:49:52 : 15/16 0.0%
19:49:56 : Test finished.

----------
Total: 15/16 (0.0%)

I just selected the beginning of the file and the difference was very clear to my ears.

You challenged me to pass such a test on AVS and I ran and reported the results both there and here.

The *important lesson* here is not that training can increase discriminatory ability -- no one in their right mind disputes that -- it's that reports of difference need to be accompanied by consideration of the *possible sources of that difference*. Saying 'some people have better hearing than others' is only part of the story, and not the most interesting here.

ABX tests do not provide any qualitative differences so you can't ask for "possible sources of difference." Nor anyone taking such tests is required to then explain the results. The listener has no such responsibility. Whoever created the tests needs to go back to the drawing board to find out why their assumption and hypothesis was wrong.

Remember, once I passed this test, people who said it was impossible also started to pass it just the same. We had poisoned the well so much with this talk of impossibility that even the people who could tell the difference, were not trying hard enough. This even included Arny himself. They are all learning to be a critical listener and ignore biased introduced to the test by folks like yourself and Arny.

Tell someone an ABX test is impossible to pass and guess what? Most people will give up immediately if the differences are small. You think double blind tests get rid of bias? Think again. We worry about listener being unbiased but then forget about the deadly sin of the experiment creator not being so.

No one here, you included, I am quite sure, actually hears the >20kHz content of the ' high rez' files. *Possibly* the bitdepth difference is audible, depending on how the conversion was done, and how abnormal the listening was (the fact that two variables -- SR and bit depth -- were changed is one of the issues making these tests problematic). Beyond that, we are left with either artifacts (from conversion, from the software , from the hardware) introduced into the audible band...or phenomena unknown to science-- and extraordinary claim require far more extraordinary evidence that provided thus far.

As I mentioned to Arny, now that these tests can be passed, then analysis can be done. Until now folks had their head in the sand thinking even more dramatic degradations such as 14 bits are inaudible as you claimed. This is why we have such a positive step forward as a result of these tests.

 

[amirm]

This would also be true in the cases where no difference was heard-thus invalidating the null results. In which case, what did you hope to achieve by putting the files up for testing

Saved me typing that .

 

[amirm]

Well to listen and see if you find areas that sound slightly different to you. To narrow down and listen intensively to short segments. If it involved an entire song of a few minutes you don't listen to whole songs. You find a segment of a handful of seconds and compare those. And your files were a good example. Even at only 12 seconds I couldn't ABX them whole. But sliding around listening to parts here or there the 2.9 second segment I settled on was discernibly different.

And this how professionally done tests of lossy compression works for example. We focus on milliseconds in a transient and not even a second worth of music.

Short term auditory memory is also just a few seconds. This is "photographic" memory where the brain stores everything that comes in and can recall it with very high accuracy. Past this, only certain aspects of the content is committed to memory as to save space. As differences get small then, it is critical to rely on short term auditory memory as to have near perfect recall. The shorter the better.

A test like Meyer and Moran where the music is playing and then the person switches to the other track completely violates this. It is fine for large differences but not small.

 

[amirm]

I just re-read Amir's posts on the first page of this thread, where he describes his methodology, and saw no reference to this. I did, however, find this:

And this:

In his initial comments, at least, Amir neither describes the procedure you outlined above or makes it sound particularl stressful. It does, however, sound as if it took some effort -- the differences to listen for had to be identified, and critical segments highlighting those differences had to be found to differentiate the samples. I'm pretty sure that's what it says, anyway. Perhaps Amir can elaborate. I'm also sure something like what you said above is in there somewhere, but even in my new-found retirement I don't have time to re-read this whole thread.

Tim

What John is saying is correct and consistent with my posts that you quoted Tim. As I have explained a few times, I listen first to A and B until I can classify their differences. I will then commit that to memory. From then on, I play X and Y for confirmation and vote. I use my memory to determine if X matched A or B. I don't play A and B again (with some rare exceptions).

The process is stressful from point of view of hitting "next trial." You get nervous because if you vote wrong, your score goes way down and you have to get a bunch right to get to small probability of guessing (I strived for zero). Even though I have classified the differences, I sometimes forget which was A or B. Another factor is reverse placebo where I can second guess myself. Controlling these factors is stressful and nothing like kicking back and enjoying the music. Think of sitting in a classroom listening to a lecture as opposed to taking a test. No matter how well you know the material, and how fast to take the exam, the anxiety is much higher in the latter.

 

[Phelonious Ponk]

What John is saying is correct and consistent with my posts that you quoted Tim. As I have explained a few times, I listen first to A and B until I can classify their differences. I will then commit that to memory. From then on, I play X and Y for confirmation and vote. I use my memory to determine if X matched A or B. I don't play A and B again (with some rare exceptions).

The process is stressful from point of view of hitting "next trial." You get nervous because if you vote wrong, your score goes way down and you have to get a bunch right to get to small probability of guessing (I strived for zero). Even though I have classified the differences, I sometimes forget which was A or B. Another factor is reverse placebo where I can second guess myself. Controlling these factors is stressful and nothing like kicking back and enjoying the music. Think of sitting in a classroom listening to a lecture as opposed to taking a test. No matter how well you know the material, and how fast to take the exam, the anxiety is much higher in the latter.

I guess I had a different understanding of what John was saying. I'm still having a bit of trouble, frankly, rationalizing the ease and speed you described in your path to the result with the great stress of attaining the result. But maybe I'm not reacting to your reports of the difficulty of the process as much as I am to how much John made of the "stress" in earlier posts in this thread.

Tim

 

[jkeny]

What John is saying is correct and consistent with my posts that you quoted Tim. ........ Think of sitting in a classroom listening to a lecture as opposed to taking a test. No matter how well you know the material, and how fast to take the exam, the anxiety is much higher in the latter.

I guess I had a different understanding of what John was saying. I'm still having a bit of trouble, frankly, rationalizing the ease and speed you described in your path to the result with the great stress of attaining the result. But maybe I'm not reacting to your reports of the difficulty of the process as much as I am to how much John made of the "stress" in earlier posts in this thread.

Tim

Wow, Tim, I envy your retirement & hope I also forget what stress means when I get there
I don't believe I made any more of it than Amir stated & my understanding led me to. I'm sure Amir would have corrected me if I gave a false impression of this, Tim.

 

[Phelonious Ponk]

Wow, Tim, I envy your retirement & hope I also forget what stress means when I get there
I don't believe I made any more of it than Amir stated & my understanding led me to. I'm sure Amir would have corrected me if I gave a false impression of this, Tim.

Retirement is awesome, John, though it is not totally stress free. As an example, today I had to decide whether I should return books to the library before going to the gym, or vice versa. I'm sure the stress completely buggered the results of the decision.

Tim

 

[jkeny]

Retirement is awesome, John, though it is not totally stress free. As an example, today I had to decide whether I should return books to the library before going to the gym, or vice versa. I'm sure the stress completely buggered the results of the decision.

Tim

Great, Tim, it's that sort of stress that can lead to a null result - did you do either ?