Rigor in tracking low level measurements involving PCs

[amirm]

Member Fiddle Faddle and I have been communicating on ways to test the Entreq. He was kind enough to send me this detailed post highlighting the measures he took to try to get reliable capture. I thought I share it here as it has good information. Repeatable measurements using a PC which capturing low-level detail is important and I enjoyed reading the techniques he used.

Please share your thoughts here. BUT DO NOT DISCUSS THE Entreq in the specific. Thank you .

---

My Entreq tests
Hi Amir,

Yes, my Entreq tests were both fraught with "hidden" difficulties. I had always imagined that even an analogue noise floor (which effectively was what I was measuring here) would be nice and consistent with a fully warmed up machine, however I found that this was only the case when a number of test parameters were extremely tightly controlled.

For example, even with the Windows 7 power profiles, small changes in CPU demands would reveal tiny, but observable changes in the noise floor. I thought I had gotten around that issue years ago by having a special "audio boot mode" (which I still use). This mode simply makes a number of (reversible) registry changes then reboots the machine. Once rebooted there are only 20 essential processes running, there is a custom power profile and a few other changes. But I found that in my Entreq testing, it was actually better to have the CPU utilisation parameters (in the power profile) set very conservatively. After much experimentation (with the sole objective being to obtain a perfectly consistent noise floor), I found that setting the minimum and maximum CPU power states to 25% seemed to achieve this. It seemed the higher I went, the higher the noise level but just as importantly, the less consistent it was - even with just 20 essential processes running. I should note that even at my "default" audio settings, the noise was still consistent with the playback card specifications - roughly 114 dB S/N ratio at 96 KHz and around 121 dB dynamic range - but just not as good or consistent as it was when throttling back on the CPU power parameters.

With that in mind, I actually did some other testing whereby I purposely started up an additional (normally unwanted) process just to see what it did to the noise floor. Sure enough, adding a CPU-hungry process did make a tiny change to the noise floor as well. At least this did arguably validate my efforts when setting up the workstation to limit the number of processes to absolutely essential ones only.

Then another interesting thing I noticed: Not only did the audio API I used make a difference to the noise floor, but so did the playback software I used! So WASAPI had a different noise floor to ASIO which was different to kernel streaming...and Foobar (for example) had a different noise floor to XXhighEnd (with XXHighEnd looking better, more consistent and less "jaggy"). In the end, I settled for ASIO playback only for a simple reason - on the playback card, it is the only API of the four available to me in Windows 7 that completely bypasses everything including the volume control. In other words, that mode was a truly "bit perfect" one - all the others - including WASAPI - would permit volume adjustments and even also DSP (such as EQ, etc).

Anyway, so in the end I did find parameters that kept the noise floor perfectly consistent for my tests, but had I not done this testing first, I would never have been able to produce a repeatable and reliable (in the circumstances and with the modest equipment at hand) result.

As for the Entreq box itself, there does seem to be an observable "rubber band" effect when connecting and disconnecting, but I did not have the time to thoroughly examine this. Nor did I consider my testing regime or equipment sufficiently good to even produce any meaningful data relating to it in any case.

In the first tests (with that "antenna" cable), the connect / disconnect results were actually ostensibly instant. Well instant in terms of I noticed the noise floor changes within the time frame needed to go to the back of the workstation, remove the cable then go back to the front of it again (so maybe 45 seconds at most). Then again, obviously there was so much noise in those first tests to begin with, if anything more subtle was happening, there was likely no way to really know.

In the second set of tests things were quite different. I actually began the second series of tests with the Entreq having already been connected for 24 hours. I was happy with the consistency of the noise floor, then ran 3 separate tests over a 15 minute period, comparing each result and making sure the results were the same (they will never be 100% identical test after test unless you are only measuring within the digital domain, but I wanted to make sure they all matched as closely as possible - and they did).

So then I disconnected the Entreq box as per usual and was "horrified" to see no change! I thought to myself - this box does nothing! (yet it had obviously made a significant difference with the terrible cable). Just as I was trying to figure out why nothing was happening, I see the noise floor begin to increase right before my eyes. I think within about 15 minutes the peaks had increased so as to be fairly consistent with my final results, but I allowed half an hour in the final test.

Not happy with this result (nothing apparently happening upon an instant disconnect), I then went to the trouble of repeating the whole test - but this time I obviously did not have the box reconnected for long - again I was "cycling" with 15 minutes on / 15 minutes off. Luckily my results were then consistent - the noise floor dropped then went up according to whether the box was connected or not. Obviously when you have an experience like that, you start questioning whether other factors are causing the noise floor fluctuations and you've just experienced a lucky (or unlucky) coincidence. But I think given how tightly I attempted to control all the other parameters (time of day - well, actually night), turning everything electrical off in the house, ensuring workstation had been warmed up for at least 12 hours prior to any testing, etc, then the results were more than likely caused by the Entreq. And given that I actually did three "cycling" tests - and each time the noise floor changed accordingly, then I think my results are reasonably robust (but again, only within the confines of the overall test - meaning a lack of an audio analyser and the lack of high quality source and recording equipment to begin with, let alone my lack of experience with serious measurement of any kind). In the end though, I actually recorded all of those noise floors then edited them in Sound Forge Pro as so as to amplify them by 100 dB or so - it was then clear the Entreq was doing "something". It was basically producing an "Entreq-specific-sounding" noise floor that was audibly different sounding to the one without once amplified so as to actually be audible.

As for how long it really does take to fully "do it's thing" (or conversely, not), I don't really know other than being fairly certain it was not necessarily instantaneous in my case - except when tested with that terrible cable. What I do feel, however, is that were I to perform any sort of rigorous Entreq testing, I'd probably be trying to measure the noise floor immediately after a disconnect, then again after 30 minutes, then 2 hours, then again after a day and then finally - if possible - after a couple of days (see more detail at the end of this PM regarding this). Again, it does not seem to matter if the equipment is actually being used during this time. Merely that the Entreq box is connected and that connection is not interrupted in any way.

And again, I did not spend much time investigating the effect of a disconnect over time. Never more than 15 - 30 minutes. And I never did any formal listening tests either like this (but there was the subjective anecdotal experience I had where I wondered why the system was not sounding its usual self one evening, only to find the cat had dislodged the Entreq cable - the connectors are their weak point. I do not like them at all). But that could have happened 24 hours prior or half an hour prior. I don't know. And it was a subjective opinion only anyway.

So here is one way I might do it, giving due consideration to the "rubber band" effect, existing users subjective feedback, Entreq guidelines as well as logistical (including time spent) concerns:


- Test rig has been warmed up for 12 hours (Entreq has never been connected before).
- Measure noise floor (and possibly a second test with a single sine wave and then a third test with two sine waves to test IM).
- Connect Entreq
- Measure immediately
- Wait 30 minutes
- Re-measure
- Wait another 90 minutes
- Re-measure
- Wait 24 hours
- Re-measure
- Disconnect Entreq
- Measure immediately
- Wait 30 minutes
- Re-measure
- Wait another 90 minutes
- Re-measure
- Wait 24 hours
- Re-measure


Maybe if with the connection tests you did one additional, final test after 48 hours, no one would have any basis to complain (this is on the assumption you could measure a change to begin with in a lesser time period). Though to be honest, if you do see any changes, I suspect nothing much will happen after 2 hours and that if it does improve over two days, that improvement will be extremely small and thus extremely hard to reliably and meaningfully measure in an analogue domain.

Anyway, I hope some of the above comments are useful to you. I don't know what boxes and cables you will test but personally speaking, if I were to measure this stuff properly, I wouldn't be wanting to teat anything less than a Silver Tellus at the minimum. By the way, the cable used makes quite an audible difference (sorry, my subjective experience only). What I don't know though, is how the cable itself influences the result versus the box itself (for example, is the noise floor just the same and it "distributes" the noise differently - in other words, each cable is a "noise shaper" and the box determines actual overall performance).

Regards

 

[Rodney Gold]

Well you first have to prove the contention that :
By the way, the cable used makes quite an audible difference (sorry, my subjective experience only)
and take it from there

 

[BE718]

OK, a lot to discuss here. A lot of things that shouldnt be happening that need to be bottomed out. I want to break it into chunks. A few questions first.

What is the sound card being used?
Is it an internal PC card?
What software do you use to take the measurements and what are its settings? ie sample rate, bit depth, window type, FFT resolution, averaging type and no. of readings (if used).

 

[esldude]

I thought I understood a good internal sound card was being used. My experience is you can't get consistent noise floors with those. Asynch USB connection to DACs and ASIO connection to ADC/DACs seems to provide a large measure of isolation to what is going on in the computer in terms of noise and software/OS differences.

Using asynch USB converters for the DAC helps by a large margin in this. I have found using ASIO devices external to the computer to be less effected. With what equipment I have to use, I have found if you have an asynch connection via USB to the DAC with a DAC that can also pass on the digital for use as a clock going into an ADC with ASIO running on a second computer gives the most consistent results. If you have to use one computer via ASIO for the ADC, it is more variable though it gives more consistent results than an internal sound card. Some ASIO devices have more consistent noise floors than others. They also are prone to performing better if they are left on for a few hours. The first hour shows some considerable noise floor improvement.

So yes, you do have to dot your I's and cross your T's.

 

[esldude]

As well as what I have already posted, I have found monitoring the clock rate if your ADC and DAC are not clock locked together is worthwhile. I typically will do measurement in one channel and send a 1/4 sample rate sine wave through the other. Some clocks are consistent, some vary over time. It seems those that vary over time have noise floor fluctuations. You won't know necessarily which clock is more accurate and whether both or one is varying. But you can see if there is a variance between the two, and how much it is.

Here is an example generated in software, but I have seen such in real measurements.



The top is the 12 khz sine at a 48 khz sample rate. The lower is the same only running fast by roughly 10 ppm. You measure the time between troughs in that waveform, multiply that by 48,000 samples per second and you know how many samples before it advances (or retards) by one sample. Allowing you to know the relative speed of the clocks. Now such a clock if consistently fast by 10 ppm may give very consistent and low noise floors. If however it speeds up and slows down over a few minutes you often will see other spurious patterns in the noise floor.

No doubt this is due to inexpensive consumer devices I have had to use. The worst I have seen is an Maudio 24/96 Audiophile PCI card. It both had high levels of jitter and the clock speed just never settled down. I have a TC Electronics IMpact Twin ADC/DAC which changes dramatically the first hour on before settling down. During this time there is a very odd rippling pattern (over about a 7 second cycle) to the noise floor around the 12 khz tone until it settles down. After it warms up it is a rather stable device.

Now most devices are pretty consistent on the clock even over days. Most will morph a bit the first hour they are on from a cold start. So in some cases when looking at very low signal or noise floor levels it is worth looking at this effect. I would suppose pro measurement devices are not going to show this very much.

So without discussing the Entreq specifically though touching on what is reported, was it the device or the measuring ADC that was changing over time when switching. If the device takes minutes to alter the noise floor that would be a very, very interesting phenomena to investigate.

 

[BE718]

Yes agreed, this is why I asked the question about internal card. Until you quantify what the actual performance of the card is, everything else is rather moot.

Internal cards are in a hostile environment. Although I Have had a RME card work well I would steer clear of internal.

Regarding all the described PC generated noise, well there is simply no point in taking measurements if there are that many problems going on, you won't be able to tell the wood for the trees. With my external adc I get extremely consistent results with no variation due to PC activity or even which PC is used. If I am not using the Intona isolator I do use 2 pcs, one for source and one for measurement as a precaution.

I simply don't buy the slowly varying noise floor,implying the entreq takes time to work. There is no mechanism for it. It is certainly nothing to do with the entreq because Fiddle clearly describes it slowly rising after disconnecting the entreq. This sounds very much like the effect of measurement averaging.

I would like to help guide Fiddle through this so he can develop some meaningful measurements, however I think he has me on ignore.

First thing for fiddle to do is get ARTA free spectrum analyser software. Terminate his sound card input with a resistor or even short it out and measure and quantify the sound card noise floor. Play with PC to see what effect it has.

If it has an effect, or the measurement shows lots of mains harmonics (like his last plots showed) then I am sorry but this internal card is unsuitable for measurement.

 

[BE718]

Below is the noise floor of my TI ADC. Below that is the noise floor af a cheap Sound Blaster X-Fi, after a few simple mods. It did suffer from a lot more noise and spuria before.

 

[jkeny]

Great to see a thread that is interested in going into measurement techniques of difficult-to-measure aspects of audio performance.
It may be that measuring these aspects are beyond the capabilities of all but experts with specialist equipment - I don't know but hopefully, discussion here might tease this out & will prove to be enlightening for all?
Look forward to the continuation of discussion.

Just one simple question - is the fact that Fiddle was aware of variation in his initial measurements & he took the time & trouble to arrange circumstances whereby he was then getting consistent sets of measurements - is this not the first stage in "proving" that his measuring system is actually measuring "cause & effect"?

Agreed that there are probably many settings that we need to find out about - such as averaging, etc which can illuminate any possible issues with the measurements

It would be good to see this thread continue in cooperation rather than aggravation.

 

[amirm]

I forgot to add , that I face similar issues when using both my Audio Precision Analyzer and Prism dSound audio analyzer. Peaks come and go constantly when I am capturing repeatedly. I write them off as too variable and only go by what is above those variations and thereby foregoing some of the sensitivity my gear has.

 

[jkeny]

I forgot to add , that I face similar issues when using both my Audio Precision Analyzer and Prism dSound audio analyzer. Peaks come and go constantly when I am capturing repeatedly. I write them off as too variable and only go by what is above those variations and thereby foregoing some of the sensitivity my gear has.

So what do you & BE (or anyone else) reckon is the best technique for measuring a noise floor to a sufficiently low level? Also any suggestions for measuring noise floor modulation at low enough levels?

 

[amirm]

Don't have any good ideas at the moment John. The noise floor modulation is one that I like to measure but just out of bandwidth to think about it.

 

[BE718]

I forgot to add , that I face similar issues when using both my Audio Precision Analyzer and Prism dSound audio analyzer. Peaks come and go constantly when I am capturing repeatedly. I write them off as too variable and only go by what is above those variations and thereby foregoing some of the sensitivity my gear has.

Just to clarify, are these longer term variations, not just random noise that will minimised with averaging? Discrete tones not wide and noise? Does it do this with The inputs terminated?

 

[jkeny]

Just to clarify, are these longer term variations, not just random noise that will minimised with averaging?

Wouldn't averaging also mask any noise modulation?

 

[BE718]

Random modulation yes, but that's not what we are looking for here. There is just random noise present, nothing you can do about that.

First thing to establish is the behaviour of the measurement system. Unless I misunderstood, Amir was talking about the noise floor without signals present.

Regarding noise floor modulation, Thinking out loud, there are two effects that I would be looking for in the noise floor - caused by out of audio band signals and in band.

I have some ideas to try and test this. I will have a play.

 

[jkeny]

Random modulation yes, but that's not what we are looking for here.

But isn't this thread about measuring the possible effects of the Entreq? Wouldn't the modulation of a noise floor just average out in an FFT as a slightly higher noise floor - i.e we would see no indication that there is any modulation going on? Just interested in this possibility

First thing to establish is the behaviour of the measurement system. Unless I misunderstood, Amir was the noise floor without signals present.

Sure, the measurement system behaviour & what's intended to be measured are inextricably linked

 

[amirm]

Just to clarify, are these longer term variations, not just random noise that will minimised with averaging? Discrete tones not wide and noise? Does it do this with The inputs terminated?

No they are sequential measurements second to second. We could get rid of them and that is what we do but I like to analyze them to find smaller effects. And they are only there from device under test.

 

[fas42]

Great to see a thread that is interested in going into measurement techniques of difficult-to-measure aspects of audio performance.

+1000

It may be that measuring these aspects are beyond the capabilities of all but experts with specialist equipment - I don't know but hopefully, discussion here might tease this out & will prove to be enlightening for all?
Look forward to the continuation of discussion.

Subjectively, I have detected this sort of thing happening for many years, . It was extremely frustrating not to have the means readily available to pick the varmints playing their little electronic games - and it was clear that no-one else had a handle on what to do about it. Even if they could afford to acquire very specialised testing gear.

I'm certain that these behaviours will be nailed - the time dependent aspect is an absolutely crucial element in all of this; as soon as one starts chasing averaging, or instantaneous, one shot readings of behaviour then this exercise will fail to find anything useful ... it will be interesting to see whether the investigators are disciplined enough to not fall into the same old traps ...

 

[BE718]

But isn't this thread about measuring the possible effects of the Entreq? Wouldn't the modulation of a noise floor just average out in an FFT as a slightly higher noise floor - i.e we would see no indication that there is any modulation going on? Just interested in this possibility

Sure, the measurement system behaviour & what's intended to be measured are inextricably linked

Fiddle is not talking about what I understand as modulation. He is describing a general change in the noise floor level over a long period of time without signals present.

Modulation is the change of the noise floor due to the presence of other signals. If what he is seeing is caused by out of band signal "interference" the change would happen immediately on removal of the entreq. It simply wouldn't take 15 minutes to change.

This is why I don't buy that the effect he is seeing had anything to with the entreq.

Until we understand exactly what he is doing, and we start using consistent tools and understand the basic performance of his sound card we are not going to get very far analysing anything his entreq is allegedly doing.

 

[jkeny]

Fiddle is not talking about what I understand as modulation. He is describing a general change in the noise floor level over a long period of time without signals present.

But Fiddle's note to Amir is just his notes on his own measurements.
It's not a prescription for the measurement limits that should be applied when measuring the Entreq

BE, have you any experience in measuring noise floor modulation?

 

[jkeny]

+1000


Subjectively, I have detected this sort of thing happening for many years, . It was extremely frustrating not to have the means readily available to pick the varmints playing their little electronic games - and it was clear that no-one else had a handle on what to do about it. Even if they could afford to acquire very specialised testing gear.

I'm certain that these behaviours will be nailed - the time dependent aspect is an absolutely crucial element in all of this; as soon as one starts chasing averaging, or instantaneous, one shot readings of behaviour then this exercise will fail to find anything useful ... it will be interesting to see whether the investigators are disciplined enough to not fall into the same old traps ...

Yes, Frank, although FFTs are a very valuable & sensitive tool, they have their limitations & recognising these limitations is worthwhile when using any tool to avoid falling into the same old traps, as you say.
A nice analogy for FFTs is a long exposure photograph - the elements that remain fixed during the exposure are sharply defined, the moving elements in the frame are blurred & indistinct. If it's the moving elements that we are interested in then long exposure photographs are not the best method to analyse or even sense these aspects exist.