WAV vs FLAC revisited

[bblue]

None of this makes it a well-controlled comparison, which is essential to rule out all sorts of psychological confounders in perception of difference and preference. What you've written up there boils down to 'I trust my ears'.

You're right. But I do trust my ears, with caution. I've been fooled a few times, but very few. And then I would examine why to understand how.

--Bill

 

[bblue]

Bill, I've never gone into the playback from computer resources, it's always been straightforward CD usage, but I'm now doing some low level experiments and getting interesting results ... see my posts on the Redbook thread.

Thanks Frank. I will check it out.

My shtick in the audio game for a long time now has been that it is all the subtle, less obvious, things that do the real damage to subjective sound quality. And one of these elements is cross interference: one piece of the electronic circuitry that's in the arrangement that produces the end sound negatively affects another. Everyone pays lip service to this, but it's hardly ever treated or dealt with seriously by the manufacturers of the gear.

Are you referring to the chain of equipment that may constitute a playback path? That any one of them could contribute negatively, which would cause additional issues downstream? If so, I would definitely agree with that. It can sometimes be much worse than just simple addition of the negatives.

So from my point of view FLAC will always suffer, or at best be equivalent, compared to WAV. Simply because more work is being done in electronic circuitry, somewhere, to get the sound from the source to the speakers.

This seems like an overly simplistic view based on unknowns, to me. The FLAC file should *always* be the equivalent of the WAV file it's made from, if the decoding is done correctly. It can't be better, though, and it shouldn't be worse. We know that a WAV file decoded from a FLAC file is identical to the WAV file that was encoded to make the FLAC file, so there's no issue with the encoding/decoding algorithm, but only something to do with the real-time decoding of the FLAC within a WAV player. It doesn't matter how much work is done in circuitry. This type of processing is trivial to the amount of work that is going on constantly through all forms of electronic circuitry.

If a cruddy MP3 can be dramatically improved in quality simply by re-sampling to hi-res then that gives the game away in that regard ...

That's not possible. MP3 is a lossy format, which means that when it is encoded (created) the audio information that is removed and which accounts for the smaller file size and lower bit rate, is gone. Forever. Not recoverable, period. No amount of re-sampling 'to hi-res' will bring back that lost information. Re-sampling may add some smoothing to the digital roughness of a low bit-rate MP3 (just like it may when up-sampling a low bit-rate WAV) but it does not recover anything that was originally lost.

--Bill

 

[sasully]

So from my point of view FLAC will always suffer, or at best be equivalent, compared to WAV. Simply because more work is being done in electronic circuitry, somewhere, to get the sound from the source to the speakers. If a cruddy MP3 can be dramatically improved in quality simply by resampling to hi-res then that gives the game away in that regard ...

Frank

You've got some fundamental misunderstandings of how these technologies work.

 

[Vincent Kars]

Uli Brueggemann did a nice experiment.
He took the original 16/44 signal and up-sampled it to 176 and then back to 44
He used SRC (Secret Rabbit Code) / cPlay

Original signal

Upsampled to 176

Downsampled to 44

Obvious resampling can have a profound impact.
It often yields audible differences
More examples can be found on the Infinite Wave website: http://src.infinitewave.ca/

http://thewelltemperedcomputer.com/KB/SRC.htm

 

[Mitchco]

FLAC vs WAV Part 2 Final Results

http://www.computeraudiophile.com/blogs/FLAC-vs-WAV-Part-2-Final-Results

 

[amirm]

mitchco, that is the best research I have seen into this claim! Well done.

The only thing I wished for , was if you had run a control through your chain. For example, capturing the same version multiple times and making sure it nulls the same way as the .wav and flac.

I will certainly bookmark this for future reference. Again, well done.

 

[Vincent Kars]

Reference material.
Best treaty on this subject known to me.

What we need is somebody doing the same test but this time measuring the jitter.

 

[garylkoh]

mitchco, that is the best research I have seen into this claim! Well done.

The only thing I wished for , was if you had run a control through your chain. For example, capturing the same version multiple times and making sure it nulls the same way as the .wav and flac.

I will certainly bookmark this for future reference. Again, well done.

+1 Thanks for the work done, mitchco.

This is my new reference when someone comes to me and tells me that according to the Absolute Sound, FLAC is inferior to WAV.

 

[Mitchco]

Thanks folks. Pretty easy to do if you want to try it yourself.

While not FLAC vs WAV... Here is a the same process used to compare two music players - JRiver vs JPLAY http://www.computeraudiophile.com/blogs/JRiver-vs-JPLAY-Test-Results

Cheers!

 

[Vincent Kars]

Hi Mitchco

The “problems” with your test are the noise level of the onboard audio and the clock drift.
Maybe it is possible to eliminate this by using a loopback recording using the Lynx
http://www.aes16.com/support_faq_result.asp?c=32

Or a simple line out / line in on the Lynx?
http://www.bvaudio.sk/Lynx/Lynx L22 Play 24_48.htm

 

[bblue]

Hi Mitchco

The “problems” with your test are the noise level of the onboard audio and the clock drift.
Maybe it is possible to eliminate this by using a loopback recording using the Lynx
http://www.aes16.com/support_faq_result.asp?c=32

Or a simple line out / line in on the Lynx?
http://www.bvaudio.sk/Lynx/Lynx L22 Play 24_48.htm

Mitcho,
I'm not buying into the conclusion of the test, as there are too many unexplained occurrences as Vincent points out.

There shouldn't be any significant clock drift if a good clock is used in the first place.

Comparing the two formats as audio rather than diigital after running the outputs through additional D/A is improper, I believe.

Neither of those two conditions would explain the uniform character difference I hear between files played real time as wav and flac. There is clearly a lack of detail when flac is played in real time, which I believe is the flac real time decoding engine. That is not an issue when you manually decode a flac to a wave file -- then the results are identical. But that is a different comparison than real-time playback.

--Bill

 

[Vincent Kars]

Nothing wrong with recording the digital out (e.g. SPDIF).
This is what is send to the DAC.
This will tell you all about the output being bit identical or not.
I do think it is a good practice to measure the digital out as it is a simple and solid technique to proof there are no differences at bit level.

However, LPCM is sample + time step.
When recording the digital out you stay in the digital domain and this won’t tell you anything about possible fluctuations in the time step.
Most of the time “improvements” in computer audio are not explained by differences at bit level but by changing the jitter (the CPlay software induced jitter concept).
The moment you leave out measuring the time part, you leave half of the LPCM phenomenon out of the equation.

Maybe clock drift is not where it is about.
It can happen but it might also be an offset problem.
If one record at 96 kHz the time step will be 0.0000104166666666667
But how to make sure each recording starts at exactly the same moment?
If the first starts at 0 and the other at 0.0000000000000000001 you sure get different samples.
Not to mention a clock running with this precision.
If a clock runs with a precision of +/- 0.000000000001 ps (a very good value I don’t expect from onboard audio) you will also have different samples in each recording.

 

[bblue]

...
Maybe clock drift is not where it is about.
It can happen but it might also be an offset problem.
If one record at 96 kHz the time step will be 0.0000104166666666667
But how to make sure each recording starts at exactly the same moment?
If the first starts at 0 and the other at 0.0000000000000000001 you sure get different samples.
Not to mention a clock running with this precision.
If a clock runs with a precision of +/- 0.000000000001 ps (a very good value I don’t expect from onboard audio) you will also have different samples in each recording.

Right, but mitchco's article describing said that you could hear the clock catching up with and overtaking the original when listening to the diff. That would be a pretty serious amount of drift, I think.

In a DAW it is pretty easy to sample align different tracks to start at the same place, and also to account for their drift as you move forward.

--Bill

 

[garylkoh]

When we tried to use DiffMaker to analyze the difference between two similar LP playback tracks, the time drift was so high that the resulting signal was almost as loud as the original. Bruce measured (I think) a time drift in the scale of ms.

Hence, if time drift is allowed for, and the result is at -90dB, I think that it's pretty impressive. What amount of time drift would result in a difference of -90dB. Is there anyway to calculate that?

Now, if we took two WAV files and did this comparison and even including time drift, the result is at say -120dB. Similarly if two FLAC files cancel to -120dB. Then we can say that the difference between WAV and FLAC is -30dB. But if the difference between two WAVs is -93dB, then the difference is -3dB. HOWEVER, can we hear the difference between -90dB and -93dB? Can we even hear the difference between -90dB and -120dB?

On the face of it, I'd say that we cannot. But there are stranger things going on in audio that we don't know how to measure.

 

[Mitchco]

Hi Mitchco

The “problems” with your test are the noise level of the onboard audio and the clock drift.
Maybe it is possible to eliminate this by using a loopback recording using the Lynx
http://www.aes16.com/support_faq_result.asp?c=32

Or a simple line out / line in on the Lynx?
http://www.bvaudio.sk/Lynx/Lynx L22 Play 24_48.htm

Hey Vincent, I included the playback DAC and analog line out stage on purpose. This is a response I just put up at Computer Audiophile as to the reason why.

The reason I wrote the FLAC vs WAV and this post was to show that my computer audio playback system is working correctly.

FLAC and WAV are lossless audio file formats, they are bit for bit identical.

Bit-perfect playback: "in audio this means that the digital output from the computer sound card is the same as the digital output from the stored audio file." http://en.wikipedia.org/wiki/Bit-perfect and "Poor device drivers often alter the data, resulting in it making not bit-perfect. This is especially true for device drivers used in consumer-grade sound cards."

If you are hearing a difference between any lossless audio file formats and/or bit perfect music players, then there is something not working correctly with your computer audio playback system (i.e. it is not bit-prefect playback).

The "free" measurement tools I presented can assist in troubleshooting what might be the issue(s).

On Windows, you can use:

DPC Latency Checker: http://www.thesycon.de/deu/latency_check.shtml DPC Latency Checker is a Windows tool that analyses the capabilities of a computer system to handle real-time data streams properly. It may help to find the cause for interruptions in real-time audio and video streams, also known as drop-outs.

To me, DPC Latency Checker is a critical tool because in my experience, a high latency computer is the number one reason where things go wrong. If you look at the latency on my computer, it is 10X below the accepted threshold. I designed my computer for this to ensure I never have any latency issues.

RightMark Audio Analyzer: http://audio.rightmark.org/index_new.shtml Excellent tool to measure the electrical noise present in your computer audio system. You can also check frequency response, distortion, etc., but it is the noise measurement is what we are mostly interested in.

Pro-tip, have a look at the size of the power supply I use in my computer. Again, in my experience, the more power, the less load = lower electrical noise. In addition, the Lynx L22 sound card has good noise rejection and a very low noise floor (-107 dB measured on my rig with DAC + ADC in external loopback mode).

Audio DIffMaker: http://www.libinst.com/Audio DiffMaker.htm Audio DiffMaker is a freeware tool set intended to help determine the absolute difference between two audio recordings, while neglecting differences due to level difference, time synchronization, or simple linear frequency responses.

I purposely included the DAC and analog line output amplifier in my tests to show that a) the Digital to Analog conversion and analog line output amplifier is not altering the bit-perfect waveform in anyway and b) the electrical noise of my playback computer is so low that I am into the noise floor on the measurement computer.

Meaning that my computer audio system is operating as it should be. Therefore, I should not hear any difference between any lossless audio file formats or bit-perfect music players.

The tools are free and the tests are simple. I encourage folks to try these tools out to ensure you are getting the best performance out of your computer audio playback system.

 

[Vincent Kars]

I don’t think bit perfect is sufficient.
Indeed the moment bits are altered it will be audible.
There are some nasty examples of Cmedia drivers doing this
http://code.google.com/p/cmediadrivers/wiki/Bitperfect

However if the bits are right, this doesn’t say anything about the timing.
You can have bit perfect output with tons of input jitter at the DAC or bit perfect out with almost zero input jitter.
Sure this makes an audible difference.
Perfect playback is bit perfect and time step perfect.

The first I do think easily to be obtained.
Best tested by recording the digital out as there is no DAC able to resolve 24 bits correctly.

The second one is impossible as there are no perfect clocks.
Our physical world simply doesn’t work that way.

Are you familiar with this blog?
http://nwavguy.blogspot.com/2011/02/jitter-does-it-matter.html

 

[Mitchco]

Hi Mitchco

The “problems” with your test are the noise level of the onboard audio and the clock drift.
Maybe it is possible to eliminate this by using a loopback recording using the Lynx
http://www.aes16.com/support_faq_result.asp?c=32

Or a simple line out / line in on the Lynx?
http://www.bvaudio.sk/Lynx/Lynx L22 Play 24_48.htm

Hi Vincent,

I used http://www.aes16.com/support_faq_result.asp?c=32 loopback capabilities and recorded Tom Petty Refugee, 24/96 FLAC/WAV using Audio DiffMaker on the same computer as the playback software.

Here is the result:
http://i1217.photobucket.com/albums/dd381/mitchatola/FLACvsWAVLynxL22Loopback.jpg

Inaudible.

With respect to sample rate drift, it occurs because in the test performed on the CA blog, I am playing back on one computer and recording, in real time, on a separate computer, without the two clocks tied together or the use of an external master clock.

With respect to the onboard sound card nose level, according to the DiffMaker FAQ:

"This is done with computer soundcards?!? Those can change the sound of music. And they are used with dirty, noisy, EMI-prone computers. And they don't have enough dynamic range or resolution.

Soundcards not being perfect, low dynamic range or resolution: You are confusing this process with A-B tests performed via ear, which are susceptible to masking effects. Masking effects are characteristics of human listening. DiffMaker tests are NOT susceptible to such masking problems, provided the resulting difference track is silent to whoever is listening to it. Diffmaker detects "really different", not merely "audibly different"."

With respect to deciding whether this is a valid test or not:

Science: http://www.libinst.com/AES Audio Differencing Paper.pdf

Slides: http://www.libinst.com/Detecting Differences (slides).pdf

Made sense to me...

With respect to jitter, yes, thanks for the articles. I have seen that bit-perfect code implementation before. Agreed. I also enjoy Bob Katz's Mastering Audio book with a chapter devoted to jitter.

I don't have a jitter measurement for the Lynx L22 card specifically. But looking at the jitter measurements on the link you provided: http://nwavguy.blogspot.com/2011/02/jitter-does-it-matter.html any of the good DAC's seem to have measurements in the -100 dBFS and lower range. Again, I would have to say inaudible relative to the program level.

http://en.wikipedia.org/wiki/Audio_file_format "Lossless compression formats enable the original uncompressed data to be recreated exactly"

Recreated exactly is the reason why I do not hear any difference between FLAC and WAV on my HTPC.

On my computer, the Audio DiffMaker results indicate that there are no significant measureable differences between FLAC vs WAV in both the digital and analog domain. Analog measuring the worst at -90 dBFS Again, relative to the program level, inaudible.

These file formats have tech specs that everyone adheres to. Same goes for bit-perfect music players. Everyone must adhere to the same technical specificatiions. So if AB testing FLAC vs WAV and it does sound different, then something is not configured/working properly on the computer. The links above provide tools to both troublehsoot and verfiy the performance of bit-perfect computer audio playback.

Cheers!

 

[amirm]

At the risk of stating the obvious, you do NOT want to sync the clocks in the capture to player. The whole point of this exercise is to see if timing of samples is changed. If DAC timing is locked to external source, then that can't happen (at least not fully).

So while this creates a drift problem in the capture scenario, it is something we have to live with.

 

[Vincent Kars]

Mitchco

Another series of very interesting exeriments.
Let check if I understand correctly all what you have done.

Comparing the recording with itself.
This tells us the maximum possible precision of DiffMaker.
This is probably the easiest case for DifMaker as it doesn’t have to correct for anything.

Comparing 2 different recordings.
Here we have a very complex situation.
- File formats are different
- Recordings are different
- Recordings are made width another computer (clock drift, etc.)
- DiffMaker has to do all kind of corrections.
This makes it hard to conclude if there is any difference between WAV and FLAC as there are so many other parameters involved.

Loopback recording on the Lynx.
This eliminates the other computer as a confounding factor.
The differences are substantial smaller.
The difference between FLAC/WAV is reduced from 90 to 162 dB but still we don’t know what the excact precision of the method is.

Mitchco
I think we need another null test.
One who eliminates as much parameters as possible.
- Using the loopback on the Lynx to eliminate the other computer as a possible source of contamination
- Record 2 times the WAV and compare to eliminated possible differences due to file formats. This will tell us the maximum possible precision of this method (given the tools used)
- Record 2 times the FLAC and compare to eliminated possible differences due to file formats. Maybe overkill but nice to know if it really equals the WAV/WAV comparison.
If WAV-WAV and FLAC/FLAC and WAV/FLAC are all equal and at an extremely low level (161) we have a far more substantial evidence than when the WAV/FLAC differences are around 90

 

[Mitchco]

Aloha Vincent, thank you for putting this summary together.

Amir says, "At the risk of stating the obvious, you do NOT want to sync the clocks in the capture to player. The whole point of this exercise is to see if timing of samples is changed. If DAC timing is locked to external source, then that can't happen (at least not fully). So while this creates a drift problem in the capture scenario, it is something we have to live with."

You say different, i.e. go with loopback...

And 1audio says: http://www.computeraudiophile.com/blogs/JRiver-vs-JPLAY-Test-Results#comment-130808

Given Audio DiffMaker software is free, I would love to see you and Amir (and others) corroborate or dispute my results with your own measurements.

Cheers!