Regarding BS1116 -- A) it's a recommendation (for subjective evaluation, by the way), not a standard. B) has anyone ever taken this recommendation? Ever? Who? In testing what? It's a nice, pretty comprehensive set of guidelines, but you'd never get audiophiles to agree on the basics -- appropriate source material and playback system.
Let me start by giving the easy answer by quoting ITU on what a recommendation is:
http://www.itu.int/pub/R-REC
The ITU-R Recommendations constitute a set of international technical standards developed by the Radiocommunication Sector (formerly CCIR) of the ITU. They are the result of studies undertaken by Radiocommunication Study Groups on:
So as you see they are "standards." Why are they called recommendations then? We get some insight from this paragraph at the end of that page:
The ITU-R Recommendations are approved by ITU Member States. Their implementation is not mandatory; however, as they are developed by experts from administrations, operators, the industry and other organizations dealing with radiocommunication matters from all over the world, they enjoy a high reputation and are implemented worldwide.
So here is the longer answer. ITU stands for International Telecommunication Union. Note the word "Telecommunication." Its job initially was to set standards that allowed interoperability between private telecommunication companies (e.g. as existed in US) and public companies (as existed in many countries encompassing much of Europe). To assure compatibility and safety of such links, ITU "standards" are literally converted to "laws" meaning breaking them can be serious offense.
Of course the world has moved on and so has ITU. We no longer have some of the issues in its original charter. No one gets electrocuted for example if a listening test is done without following all the components of BS1116.
Let's look at a better example. Anytime you turn on your HDTV, the picture you see is governed by another "Recommendation" from ITU called BT709. The "t" stands for Television by the way and "S" in BS1116 stands for "sound." The context of both is for broadcasting. But this being a recommendation, it is not mandatory and indeed some DVD players violated it. Recall when DVD players started to have "upsampling" where they would take the DVD and output high definition video? The DVD like broadcasting of non-HD content was governed by another ITU standard called BT601. BT601 must not be used with HDTV as it has somewhat different color than BT709. Not realizing this, some manufacturers output HDTV but kept the color format BT601. The TV however, would switch to BT709 anytime it received HD signal and as a result, display incorrect colors. The right solution was to not only upsample the pixels but also perform the color conversion to BT709.
So while ITU recommendations are followed to a letter in many instances, it not being the law means that at the extreme, you have no obligation to follow them as such. But try to sell any broadcaster equipment for HDTV that doesn't comply with ITU-RBT709 and even the receptionist at the front door of a major network would know to throw you out of the building
. "Optional" does not have the lay meaning that we assume here.
It is being held up here as a standard, without which any result can be dismissed. Where has it been used?
Tim
I didn't hold it up as a "standard" but rather, best practices. Please go back and read my posts where I use this term and BS1116 interchangeably. Normally I would not even reference BS1116. But this group constantly wants validation so I am doing that to show these are not my ideas.
I already quoted one paper that used ITU-R BS1116 in test of DVD-A against SACD. Here is a number of others which you can also find yourself by searching the AES web site:
P20-1 Selection of Audio Stimuli for Listening Tests—Jonas Ekeroot, Jan Berg, Arne Nykänen, Luleå University of Technology - Luleå, Sweden
Two listening test methods in common use for the subjective assessment of audio quality are the ITU-R recommendations BS.1116-1 for small impairments and BS.1534-1 (MUSHRA) for intermediate quality. They stipulate the usage of only critical audio stimuli (BS.1116-1) to reveal differences among systems under test, or critical audio stimuli that represents typical audio material in a specific application context (MUSHRA). A poor selection of stimuli can cause experimental insensitivity and introduce bias, leading to inconclusive results. At the same time this selection process is time-consuming and labor-intensive, and is difficult to conduct in a systematic way. This paper reviews and discusses the selection of audio stimuli in listening test-related studies.
Convention Paper 8445 (Purchase now)
7-1 Subjective Evaluation of Large and Small Impairments in Audio Codecs
Gilbert A. Soulodre, Michel Lavoie, Advanced Sound Systems Group Communications Research Centre, Ottawa, Ont., Canada (Invited)
A significant amount of research has been conducted in the development of a test methodology for evaluating small impairments in audio codecs. Specifically, ITU-R Recommendation BS.1116 provides a full description of the accepted methodology which has been used extensively in the development and evaluation of high quality audio codecs. This methodology has proven effective at generating consistent results in subjective tests and can provide a high degree of resolution in discriminating between codecs. With the recent trend towards very low bitrate audio codecs and the corresponding lower quality, a need has arisen for a subjective test methodology which will allow the performance of these codecs to be evaluated and compared in a rigorous fashion. The method described in BS.1116 is not entirely appropriate for this purpose and so a new method has been developed for evaluating audio codecs with larger impairments. The new method strives to maintain those aspects of BS.1116 which have proven most effective, while extending it to address the particular difficulties encountered when evaluating large impairments. This paper describes the methodologies for evaluating small and large impairments, and includes results from formal subjective tests.
Evaluation of the ITU-R Objective Audio Quality Measurement Method
William C. Treurniet and Gilbert A. Soulodre 164
The subject of this study was the assessment of the PEAQ (perceptual evaluation of audio quality) method's ability to predict and correlate the results of subjective tests conducted according to ITU-R BS-1116. An additional goal was to determine PEAQ's ability to correctly rank the performance quality of various codecs. The PEAQ system was also evaluated as an aid to selecting appropriate audio listening materials for use in subjective listening tests.
P10-7 Evaluation of HE-AAC, AC-3, and E-AC-3 Codecs—Leslie Gaston, Richard Sanders, University of Colorado at Denver and Health Sciences Center - Denver, CO, USA
The Recording Arts Program at the University of Colorado at Denver and Health Sciences Center (UCDHSC) performed an independent evaluation of three audio codecs: Dolby Digital (AC-3 at 384 kbps), Advanced Audio Coding Plus (HE-AAC at 160 kbps), and Dolby Digital Plus (E-AC-3 at 224 and 200 kbps). UCDHSC performed double-blind listening tests during the summer of 2006, which adhered to the standards of ITU-R BS.1116 (that provides guidelines for multichannel critical listening tests). The results of this test illustrate a clear delineation between the AC-3 codec and the others tested. We will present the test procedures and findings in this paper.
There are many more. But perhaps more familiar to this audience is our own Dr. Sean Olive and Dr. Floyd Toole with this paper: A New Laboratory for Evaluating Multichannel Audio Components and Systems, Authors: Olive, Sean E.; Castro, Brian; Toole, Floyd E.
The current standards that recommend listening room performance include: ... ITU-R Recommendation BS. 1116: Methods for Subjective Evaluation of Small Impairments in audio systems including multichannel sound systems, 2.d Edition (1997) [12]
As you see, in the industry/research world BS1116 is extremely well known and utilized. Can you deviate from it? Sure. But you better have a good reason. In my case I used headphones and not a room with reverb time, etc. as they recommend. This however is an accepted deviation. Throwing out such things as critical audio test segments, controls, pre-screening and such is simply not done in properly done tests even if it is not referencing BS1116. We know from experience that it is hugely simple to get negative results by creating less sensitive tests, and inclusion of many people with non-critical listening abilities.
I will stop here as I suspect you did not remotely expect an essay for your simple question
.
