A question for Mr. Science....

[Phelonious Ponk]

This is pretty simple, I hope I get it right:

It has been proposed to me that no AB/X study can be valid without a hypothesis, that further, to be valid the hypothesis must be tested against a null hypothesis and that this...

This study aims at investigating whether listeners can perceive differences between musical files recorded at 44.1kHz and 88.2kHz with the same analog chain and type of AD-converter.

...is a hypothesis. I'm struggling. What would be the null of a neutral?

Tim

PS: to be clear, I have no problem believing that yes, listeners can perceive that particular difference, so that's not the issue here.

 

[fas42]

Well, what you call a hypothesis is not really that, it has to be shortened, as in the bold bit:

This study aims at investigating whether listeners can perceive differences between musical files recorded at 44.1kHz and 88.2kHz with the same analog chain and type of AD-converter.

The corresponding null hypothesis would be:

"listeners can't perceive differences between musical files recorded at 44.1kHz and 88.2kHz with the same analog chain and type of AD-converter"

Frank

 

[Phelonious Ponk]

I'm not the one calling it a hypothesis and your answer is exactly what I expected. To be a hypothesis, it has to predict the result, and the null hypothesis would predict the opposite outcome. The whether, the neutrality, has to come out.

So what about the other part of the question: Would testing, AB/X or otherwise, require a hypothesis, to be tested against a null, to be valid? I'm not scientist, so it is not a trick question, but my understanding has always been the opposite -- when doing this kind of research you specifically do not want to predict a result. You want to design the test from a neutral perspective if at all possible.

Yes? Or am I confused?

Tim

 

[JackD201]

Boy, it's been a long time. I believe you go with the hypothesis as it is really a matter of form. The reason for the test is to see if results whether conclusive or anything less than that are repeatable. Controlled conditions follow from that logic. Only then can conclusions be drawn.

If you think about it in layman's terms, there's nothing to prompt the formation of a hypothesis if there wasn't anything observed in the first place. No observation, no question, no hypothesis, nothing to prove or disprove. It doesn't matter if the observation is suspected to be imagined or not. The test if done properly should make that determination, at least statistically.

 

[rbbert]

As previously noted, the answers are here

http://www.whatsbestforum.com/showthread.php?6340-Perception-science-engineering-and-high-end-audio

And as I've tried to explain, forming a hypothesis is not predicting a result, otherewise there's no reason to test; the hypothesis is the question.

 

[fas42]

Part of the answer is not to confuse the ABX testing, with the hypotheses, null or otherwise. They are two separate processes or activities: the first is to acquire experimental data, obtained in as neutral a manner as possible. The formulation of the hypotheses is something that can be done before, during or after the testing; in other words it is an independent process. The two join up at the point where the experimental data is analysed, to see whether it supports, statistically, the hypothesis, or, the null hypothesis.

Frank

 

[fas42]

As previously noted, the answers are here

http://www.whatsbestforum.com/showthread.php?6340-Perception-science-engineering-and-high-end-audio

And as I've tried to explain, forming a hypothesis is not predicting a result, otherewise there's no reason to test; the hypothesis is the question.

This is more along the lines of being aware of a timeline in the independent processes of testing, and hypothesising. So one timeline may be that someone formulates a hypothesis, out of thin air, so to speak. He then devises a series of tests in an attempt to validate his assertion. With an end result that his hypothesis may be supported, or it may not.

Or, another timeline could be that a phenomenon is noted, and a whole lot of data is collected, or a set of tests is devised to systematically capture aspects of the behaviour, without understanding it. Then, down the track a hypothesis is formulated, in a flash of inspiration by someone, and then that "old" data is then reprocessed, so to speak, to see if the results support the hypothesis.

Frank

 

[JackD201]

^
They said it better.

 

[Phelonious Ponk]

As previously noted, the answers are here

http://www.whatsbestforum.com/showthread.php?6340-Perception-science-engineering-and-high-end-audio

And as I've tried to explain, forming a hypothesis is not predicting a result, otherewise there's no reason to test; the hypothesis is the question.

You have tried to explain it, but you haven't gotten the job done, unfortunately. If it must be shortened, as Frank proposes, to be a hypothesis...

listeners can perceive differences between musical files recorded at 44.1kHz and 88.2kHz with the same analog chain and type of AD-converter.

...then you have made a prediction (that they can hear a difference) and you are testing an assumption. If the statement stands as it is, in this study and most AB/X studies I've seen...

This study aims at investigating whether listeners can perceive differences between musical files recorded at 44.1kHz and 88.2kHz with the same analog chain and type of AD-converter.

...there is no assumption, no prediction and there can be no null hypothesis because there is no opposite to a neutral.

I think Jack has it right:

If you think about it in layman's terms, there's nothing to prompt the formation of a hypothesis if there wasn't anything observed in the first place. No observation, no question, no hypothesis, nothing to prove or disprove. It doesn't matter if the observation is suspected to be imagined or not. The test if done properly should make that determination, at least statistically.

No observation, no information gathered, nothing to prove or disprove -- no hypothesis. This means, of course than an AB/X test of this kind is a gathering of statistical data, not a scientific test. And while it may strongly indicate an outcome to be hypothesized ( is that a word?), it isn't proof.

But I've never believed it was any more than that, so I'm good. Can anyone else shed anymore light on this?

Tim

 

[Phelonious Ponk]

Or, alternately, what Frank said (mark this moment).

Tim

 

[JackD201]

Hi Tim,

Like you I have a marketing background so we've both relied heavily on statistics. This isn't to say that statistics isn't scientific. One just can't bandy about conclusions the way one could with, say, geometric proofs.

My favorite is "Mammals don't lay eggs". Statistically it would be pretty darned true enough until Frank sends you a Platypus. There goes the neighborhood. LOL.

True AB/X can be and is an extremely useful tool. I do agree that you can't do an AB/X without a hypothesis. All you'd have is a bunch of guys sitting around writing random stuff while someone works the switch box. If the guy handling the box asks even one question, THAT becomes the hypothesis in question form.

 

[Phelonious Ponk]

Hi Tim,

Like you I have a marketing background so we've both relied heavily on statistics. This isn't to say that statistics isn't scientific. One just can't bandy about conclusions the way one could with, say, geometric proofs.

My favorite is "Mammals don't lay eggs". Statistically it would be pretty darned true enough until Frank sends you a Platypus. There goes the neighborhood. LOL.

True AB/X can be and is an extremely useful tool. I do agree that you can't do an AB/X without a hypothesis. All you'd have is a bunch of guys sitting around writing random stuff while someone works the switch box. If the guy handling the box asks even one question, THAT becomes the hypothesis in question form.

So "Is it A or B" is a hypothesis? What's the null hypothesis?

Tim

 

[JackD201]

Somehow I don't see that question being asked. Of course it could be, I just don't know what a researcher could get out of that unless he's testing for some very gross differences. Even that would be a hypothesis.

 

[rbbert]

Nope, Frank got it wrong. As the scientific method has clearly stated for over 300 years, first one gathers data, and from that data you generate a hypothesis (question). Certainly one can ask the question after the data is gathered in order to allow for statistical analysis of that data, BUT THE ONLY CONCLUSION APPROPRIATE FROM THAT IS THE GENERATION OF A HYPOTHESIS FOR TESTING. The scientific method requires that the question (hypothesis) be generated from the data empirically collected. A study design is then developed to test the hypothesis, data collected and then analyzed. THEN the results may allow for a valid conclusion to be drawn, but not before.

 

[Phelonious Ponk]

Nope, Frank got it wrong. As the scientific method has clearly stated for over 300 years, first one gathers data, and from that data you generate a hypothesis (question). Certainly one can ask the question after the data is gathered in order to allow for statistical analysis of that data, BUT THE ONLY CONCLUSION APPROPRIATE FROM THAT IS THE GENERATION OF A HYPOTHESIS FOR TESTING. The scientific method requires that the question (hypothesis) be generated from the data empirically collected. A study design is then developed to test the hypothesis, data collected and then analyzed. THEN the results may allow for a valid conclusion to be drawn, but not before.

OK...I'm ok with it if I don't get it. Help me. Here is the summary statement from the study you linked me to in our offline conversation...

It is currently common practice for sound engineers to record digital music using high-resolution formats, and then down sample the files to 44.1kHz for commercial release. This study aims at investigating whether listeners can perceive differences between musical files recorded at 44.1kHz and 88.2kHz with the same analog chain and type of AD-converter. Sixteen expert listeners were asked to compare 3 versions (44.1kHz, 88.2kHz and the 88.2kHz version down-sampled to 44.1kHz) of 5 musical excerpts in a blind ABX task. Overall, participants were able to discriminate between files recorded at 88.2kHz and their 44.1kHz down-sampled version. Furthermore, for the orchestral excerpt, they were able to discriminate between files recorded at 88.2kHz and files recorded at 44.1kHz

The bold-faced sentence is the one you said was the hypothesis. You also said that such a study couldn't conclude anything without the hypothesis being tested against a null. I'm with you up to this point....

What's the null of "This study aims at investigating whether listeners can perceive differences between musical files recorded at 44.1kHz and 88.2kHz with the same analog chain and type of AD-converter?"

Tim

 

[rbbert]

Hypothesis being tested: Listeners can perceive differences. Null hypothesis: Listeners cannot perceive differences. The statistical approach to the two questions is slightly different. however, the somewhat imprecise wording is common in the scientific literature; "whether or not" is commonly used even though the test is clearly to determine "whether", not "or not".

 

[JackD201]

Frank was right in deleting "This study aims to investigate whether" from the bold faced selection. The tradition is to state the hypothesis in the affirmative. Statement in the negative is the null hypothesis.

I'm not to sure about gathering data always being the first step here unless even a casual observation could be considered as a gathering of data. Something has got to provide the impetus for the data gathering process unless the process was begun to find out something else and a pattern was observed that led to another hypothesis. Sort of like the ancients mapping lunar cycles for agricultural purposes and they happened to notice that tidal cycles appeared to be correlated to the lunar. It all still started with them saying hey, the moon looks different, how many nights till it looks like this again?

 

[Phelonious Ponk]

Hypothesis being tested: Listeners can perceive differences. Null hypothesis: Listeners cannot perceive differences. The statistical approach to the two questions is slightly different. however, the somewhat imprecise wording is common in the scientific literature; "whether or not" is commonly used even though the test is clearly to determine "whether", not "or not".

If wording that imprecise is common then the literature isn't worth much. What in that statement quoted above made you think the hypothesis was "Listeners can perceived differences" instead of "Listeners cannot perceive differences?" I'm no scientist, but my language and logic skills are fairly sound. That statement doesn't commit. "Whether" makes it a completely different statement from "listeners can perceive differences."

Perhaps they're just intending to gather data to be hypothesized upon? Or perhaps they really, really suck at constructing a simple sentence. But your assumption is too much. The statement doesn't say what you're attributing to it. And you don't have anything else to base your assumption on, because neither of us have read the rest of the paper.

Tim

 

[Phelonious Ponk]

I'm not to sure about gathering data always being the first step here unless even a casual observation could be considered as a gathering of data.

I'm not sure either, Jack. But I am sure that statement doesn't presume that listeners can hear the difference. I believe they can, FWIW, that's not the point.

Tim

 

[rbbert]

If wording that imprecise is common then the literature isn't worth much. What in that statement quoted above made you think the hypothesis was "Listeners can perceived differences" instead of "Listeners cannot perceive differences?" I'm no scientist, but my language and logic skills are fairly sound. That statement doesn't commit. "Whether" makes it a completely different statement from "listeners can perceive differences."

Perhaps they're just intending to gather data to be hypothesized upon? Or perhaps they really, really suck at constructing a simple sentence. But your assumption is too much. The statement doesn't say what you're attributing to it. And you don't have anything else to base your assumption on, because neither of us have read the rest of the paper.

Tim

The stastical analysis used adressed the hypothesis/supposition/question "Listeners can perceive differences"; it did not address "listeners cannot perceive differences" or "can listeners perceive differences or not".