FFT audio analysis flaws?

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
I have to thank Vincent Kars for linking me to Jim LeSurf's site a while back. In it I found many interesting pieces of information among which was the IQ-test that Vincent linked me to http://www.audiomisc.co.uk/Linux/Sound3/TimeForChange.html.

In summary, the IQ-test is Jim's replacement for the ubiquitous J-test, which is fine for SPDIF signals but mis-applied when testing USB audio signals). In other words Jim found that the J-Test was inappropriate for testing USB audio signals as it was designed for exposing the flaws in SPDIF signalling. His IQ-test was able to uncover the digital "wow & flutter" of USB audio i.e differences in timing over a longer analysis period than the usual analysis period that is used for testing jitter. I will come back to this in a minute & ask some questions about it. But first some background:

The explanation of what the IQ-test does & the thinking behind it is given here: http://www.audiomisc.co.uk/Linux/Sound3/TheIQTest.html
The source code for anyone wishing to run the test is given here: http://www.audiomisc.co.uk/software/index.html scroll down to near the bottom
IQ-test results for the DACMagic & Halide Bridge are given here http://www.audiomisc.co.uk/Linux/Sound3/TimeForChange.html and also an explanation for the test setup.

From that first IQ-test the following graphs are generated: This IQ-test is a test at the analogue outs of the DACmagic when fed audio via it's native adaptive USB input Vs the asynchronous USB Halide Bridge feeding SPDIF into the DACMagic
Fig2.jpg
The blue line shows the results for USB via the Halide Bridge. The red line shows the results for a direct USB connection into the DACMagic. In this case the test was using ‘CD standard’ LPCM data – i.e. a 44·1k sample rate using 16 bit values. By comparing the two replay methods you can clearly see that the direct USB connection produces periodic ’jumps’ in the replay speed. These don’t occur when sending the data via the Halide Design USB-SPDIF Bridge. The Bridge produces a much smoother and more regular replay of the data.

None of this is terribly surprising, perhaps, when comparing adaptive USB to asynchronous USB. However the extent of the difference on the analogue outs of the DACMagic is probably surprising given the ASRC that it uses - "Our unique ATF2 (Adaptive Time Filtering) upsampling technology, developed in conjunction with Anagram Technologies, Switzerland." This may give rise to the answer - well that low frequency jitter seen in the IQ-test will get through any ASRC because it's below most if not all rate estimator's cut-off frequency.

Feel free to discuss this point if wished but here's the bit I'm most interested in asking about. Given that we have FFT analysis on Stereophile of the DACMagic jitter at it's analogue outs, why is this gross level of jitter not showing up in the FFTs?
By gross level jitter, I mean that in that IQ-test graph the timing jumps by 1micro Sec & stays there for a duration of 2 secs before dropping back to the original correct timing. This pattern is repeated every 10 seconds. Each individual jump is not big but when you translate this into jitter figures it is significant - according to LeSurf's analysis of the 48KHz test which shows a jump of 8 micro secs "Here the rate jumps down about 8 ppm for around 2 seconds at a time. Now an 8 ppm change in rate accumulates to a timing error of 16 microseconds over two seconds. i.e. a ‘jitter’ over this period of 16 million picoseconds! This is many orders of magnitude greater than the kinds of values reported for J-Test measurements on shorter timescales!"

When I considered why this gross jitter level would not show up as distortion on a standard audio FFT, I initially thought that it would be seen as a flattening of the central peak as it would be very close in jitter - much like very close-in phase noise shows as a widening of the base of the carrier frequency spur in an FFT.

Then I considered & chatted & thought that a standard FFT as used in audio (Stereophile) would probably not pick up this distortion anyway because of the construction of the test. Am I right in thinking that typically, for 16/44 samplerates using 1KHz test tone, an analysis of about 1 sec of audio is all that's considered necessary for "full FFT analysis". This may be repeated 4, 8, 16 or 32 times for averaging by an AP. This 1 sec sample window would likely miss the jump in timing that occurs every 10 secs. Even if the FFT sample window was 10 secs would it actually show the resulting distortions in it's plot? To my simplistic understanding of FFT operation, it will treat random events (this jump) as noise & it will become buried in the grass at the bottom of the FFT plot.

Any experts wish to correct or discuss my musings?
 
Last edited:

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
This is the Stereophile FFT for the DACMagic USB input at 16/44kHz, same as the IQ_test
509Camfig12.jpg

So are these close-in spurs seen in the graph, the result of distortions from the jitter shown in the IQ-test?
Most objectivists would say any distortion spurs below -100dB as is shown on this FFT is not audible.
Yet, almost all who have heard the DACMagic (blind testing, level matached, etc) report the USB input as sounding substantially worse than other inputs on the DACmagic
 

DonH50

Member Sponsor & WBF Technical Expert
Jun 22, 2010
3,947
306
1,670
Monument, CO
I suspect the Stereophile results do not show the results of those glitches in the other test but don't know. Close-in spurs could be caused by clock jitter/phase noise as well as other things like distortion aliasing back in-band.

The record length and windowing functions are among the Achille's heels of FFT analysis. Related is their ability to deal with a complex time-variant signal, such as (ahem) music. I would love to see more multi-tone and NPR test results.
 
Last edited:

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
I suspect the Stereophile results do not show the results of those glitches in the other test but don't know. Close-in spurs could be caused by clock jitter/phase noise as well as other things like distortion aliasing back in-band.
Ok. I suspect that the Stereophile FFT was run on 1 sec audio samples & maybe repeated up to 32 times for the averaging necessary in FFTs. What sort of bin size would be needed to show up the IQ-test "glitches" in a FFT run on a 10 second audio sample using 3150Hz (14 samples per cycle) test signal frequency @ 44.1KHz sample rate?

The record length and windowing functions are among the Achille's heels of FFT analysis. Related is their ability to deal with a complex time-variant signal, such as (ahem) music. I would love to see more multi-tone and NPR test results.
indeed but even with sine wave testing we can see that the typical FFT run in audio is not sensitive enough! So therefore claims using FFT distortions as "proof" of inaudibility are based on a "belief system".
 

DonH50

Member Sponsor & WBF Technical Expert
Jun 22, 2010
3,947
306
1,670
Monument, CO
I do not know what Stereophile uses for testing (do not have time to look for it now).

To grab 15 s you'd need 44.1e3*15 = 662,000 samples, a fairly large value. FFTs of 32K to 64K are routine, with 1M-pt FFTs not out of line. 1M points yields about 22 seconds of data. The noise floor scales with the number of samples but you shouldn't need a huge record; the problem lies in capturing and presenting time-variant inputs. FFTs usually assume (but do not hve to) a time-invariant system. Sine waves are relatively easy.

I am still not sure what the rate error plot shows -- 1 ppm error of a 44.1 kS/s system corresponds to a time error of about 23 ps, not overly large. The glitches appear to be about every 15 s, which should generate a spur at 1/15 s = 0.0667 Hz, well outside the FFT window shown. Assuming it modulates the clock and/or signal, you'd expect to see spur skirts around the clock and any signal frequencies, but that's a very low frequency.

I could try to simulate a test case this weekend.
 

Kal Rubinson

Well-Known Member
May 4, 2010
2,360
697
1,700
NYC
www.stereophile.com
So therefore claims using FFT distortions as "proof" of inaudibility are based on a "belief system".
Glad you put proof in quotes since no measurement can prove inaudibility. One can only infer that from measurements, regardless of your belief system.
 

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
Thanks Don but I don't think that there is any need, unless you are interested in developing a test case.

Here's the same measurements as graph one I posted but plotted in the more familiar format of a frequency spectrum:
Fig3.jpg

Figure 3 shows the same results, but this time in the form of a spectrum of the ‘flutter’ in the replay rates. Again the blue line is when the Halide Bridge is used, and the red is for a direct USB connection. The forest of red spikes below about 1·5Hz is the spectrum of the periodic ‘jumps’ shown in Figure 2.

So these jumps are giving rise to low frequency (below 1.5Hz) distortion to a maximum of 300 parts per billion. The interesting question is - does this measurement give a correlation to sonic performance in this instance or is it a red herring? The DACMagic native USB input is reported to sound less good than it's other inputs (interested to hear from owners to confirm this) & less good than when using the Halide Bridge as the USB receiver to SPDIF transport. I guess other factors come into play i.e the whole SPDIF signal path being different to the USB signal path within the DACmagic.
 

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
Glad you put proof in quotes since no measurement can prove inaudibility. One can only infer that from measurements, regardless of your belief system.
Yes, agreed but I've come across some of the more rabid objectivists who have this belief that if all distortions shown on the FFT are below, lets say -100dB, then the device is audibly transparent.

BTW, Kal, have you used the IQ-test yourself? Any results to share?
 

Kal Rubinson

Well-Known Member
May 4, 2010
2,360
697
1,700
NYC
www.stereophile.com
Yes, agreed but I've come across some of the more rabid objectivists who have this belief that if all distortions shown on the FFT are below, lets say -100dB, then the device is audibly transparent.

BTW, Kal, have you used the IQ-test yourself? Any results to share?
I have done no testing, other than acoustics/room measurement, since I do not have the requisite instruments. Just an observer.
BTW, the more rabid subjectivists are equally guilty of using any measured distortions as "proof" of audibility, again without any measures of audibility, per se.
 

opus111

Banned
Feb 10, 2012
1,286
3
0
Hangzhou, China
The DACMagic native USB input is reported to sound less good than it's other inputs (interested to hear from owners to confirm this) & less good than when using the Halide Bridge as the USB receiver to SPDIF transport. I guess other factors come into play i.e the whole SPDIF signal path being different to the USB signal path within the DACmagic.

I'd speculate using my usual suspect of common-mode noise being the reason - USB sources are typically PCs which are much more noisy, probably the designers haven't isolated the USB in? Or if they have isolated its only decent isolation at DC.
 

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
I have done no testing, other than acoustics/room measurement, since I do not have the requisite instruments. Just an observer.
It would be great to see some additional measurements done using this test - an evaluation of it's validity to audibility might emerge?
BTW, the more rabid subjectivists are equally guilty of using any measured distortions as "proof" of audibility, again without any measures of audibility, per se.
True & I hope that this thread may help to prevent that by discussing/investigating these measurements/test.

Edit: In regard to the USB in of the DACMagic, Lesurf seemed to say that using the Halide Bridge was an improvement in sound, Stereophile had this to say about it "Although its USB input is really of only utility quality and shouldn't be used for serious listening,"
 
Last edited:

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
1.5 Hz out of 3500 Hz is not many cents if you are talking music...

Sure, in the analogue world, this would equate to wow of 0.04%.
Some questions then:
- is this "digital wow" analogous to analogue wow? Akmirm said this in his small signal distortions thread
Masking says that wow flutter at low frequencies such as a few Hertz, has to be inaudible per my article posted here. It simply is too close to the original tone to be audible. Yet, it is clearly audible. The reason it is audible is because its amount is very high -- orders of magnitude higher than jitter in digital systems. At those levels, it starts to change the volume/envelop of the signal. The ear is able to detect volume changes if they are at certain rates. Too slow and it adapts to them and can't hear the change. Too high and it can't hear them. 4 Hz is the peak detection.

For these reasons, there is little we can learn from application of wow and flutter to digital systems. The notion that large amounts of them can be inaudible in analog systems does not at all instruct us to think that large amounts of jitter in digital systems will act the same way. Both frequency and levels are different for the two universes.

- "Professional tape machines can achieve a weighted flutter figure of 0.03%, which is considered inaudible, but for the fact that without weighting it would be an actual 0.3%." Is this relevant to the above .04% "digital wow"?

- Is this any different to clock wander or the close-in phase noise of clocks? This sort of phase noise is said to be particularly detrimental to sound.
 

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
I'd speculate using my usual suspect of common-mode noise being the reason - USB sources are typically PCs which are much more noisy, probably the designers haven't isolated the USB in? Or if they have isolated its only decent isolation at DC.

Are you thinking that this IQ-test reveals said CM noise or are you saying that it's a separate issue to what is being displayed on the IQ_test plots?
 

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
Its my best guess for the reported difference in SQ - its orthogonal to the results of the IQ_test in that it would still exist if the USB were async.

I'm not sure I understand the orthogonality concept here as you have said that even if the DACmagics USB in was asynchronous the CM noise would still exist. However, if the USB in was asynchronous the plot produced in IQ-test would be much more like the HAlide Bridge & we wouldn't be seeing the glitches that we see on the DACMagic plot.

Here's the only other IQ_test plot I know of - this time done on another asynchronous device, the Arcam rDac
Fig1_44k_d_16bit.jpg
The X time division scale on this plot is at a much finer detail than the Halide Bridge but you can see that in general this plot is like the Halide Bridge plot.
 

opus111

Banned
Feb 10, 2012
1,286
3
0
Hangzhou, China
I'm not sure I understand the orthogonality concept here as you have said that even if the DACmagics USB in was asynchronous the CM noise would still exist.

Yep that's what I mean by them being orthogonal - effects independent of one another.

However, if the USB in was asynchronous the plot produced in IQ-test would be much more like the HAlide Bridge & we wouldn't be seeing the glitches that we see on the DACMagic plot.

Indeed not - but my conjecture is that the SQ difference reported would still exist. Fixing that first, we might find that the digital wow produced its own audible effect, but I'm betting that's currently masked by the CM noise effect.
 

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
OK, thanks for the explanation
 

DonH50

Member Sponsor & WBF Technical Expert
Jun 22, 2010
3,947
306
1,670
Monument, CO
A quick jump in and out: The speed change shown appears to be a "jump" that would produce broad-band noise since it is a nonlinear thing, not quite the same as more linear'ish wow and flutter that tends to be a continuous signal. Wow and flutter might add tones like close-in phase noise. The speed change "jump" may also add close-in tones (skirts), but probably also burts of wideband noise given it is jumping discretely.

I tend to agree with opus111 that there are other reasons for hearing differences and suspect this (speed/rate jump) effect is not one of them. I seriously doubt we can pull a 0.1-Hz (or whatever) tone out when it is masked by the signal tones and well below audibility.

CM noise across the USB interface can be a problem for asynch or synchronous -- it can creep into the analog output stages as well as corrupt the digital bitstream.
 

jkeny

Industry Expert, Member Sponsor
Feb 9, 2012
3,374
42
383
Ireland
A quick jump in and out: The speed change shown appears to be a "jump" that would produce broad-band noise since it is a nonlinear thing, not quite the same as more linear'ish wow and flutter that tends to be a continuous signal. Wow and flutter might add tones like close-in phase noise. The speed change "jump" may also add close-in tones (skirts), but probably also burts of wideband noise given it is jumping discretely.

I tend to agree with opus111 that there are other reasons for hearing differences and suspect this (speed/rate jump) effect is not one of them. I seriously doubt we can pull a 0.1-Hz (or whatever) tone out when it is masked by the signal tones and well below audibility.

CM noise across the USB interface can be a problem for asynch or synchronous -- it can creep into the analog output stages as well as corrupt the digital bitstream.

I'm still trying to get my head around this. Donh, I'm not sure about this issue of the close-in phase noise being difficult to hear? It seems that there is a lot of indications that close-in phase noise (<10Hz) is crucial to audio. I wonder if it is because it directly effects the central frequency (frequency stability) causing some sort of smear, rather than higher frequency jitter resulting in outlying spurs causing some sort of tone/harmonic that is distant from the central frequency? Can this be explained by existing psychoacoustics - I know that psychoacoustic masking says that it should be less obvious so I can't fathom how close-in phase noise (or LF jitter) is said to have such an audible effect. from people who seem knowledgeable in this area. Some saying "We found that going down to 0.1 Hz was not low enough." but I'm not sure this isn't a bit of machismo?

However what seems to be that Allan deviation (or variance) = 0.1 sec to 1000 sec (or longer) - as an indicator of crystal quality in audio may not be a bad yardstick.

Is this IQ-test a way of measuring such Allan variance in circuit?
 

About us

  • What’s Best Forum is THE forum for high end audio, product reviews, advice and sharing experiences on the best of everything else. This is THE place where audiophiles and audio companies discuss vintage, contemporary and new audio products, music servers, music streamers, computer audio, digital-to-analog converters, turntables, phono stages, cartridges, reel-to-reel tape machines, speakers, headphones and tube and solid-state amplification. Founded in 2010 What’s Best Forum invites intelligent and courteous people of all interests and backgrounds to describe and discuss the best of everything. From beginners to life-long hobbyists to industry professionals, we enjoy learning about new things and meeting new people, and participating in spirited debates.

Quick Navigation

User Menu

Steve Williams
Site Founder | Site Owner | Administrator
Ron Resnick
Site Co-Owner | Administrator
Julian (The Fixer)
Website Build | Marketing Managersing