The Misinformed Misleading the Uninformed -- A Bit About Blind Listening Tests

Ron Party

WBF Founding Member
Apr 30, 2010
2,457
13
0
Oakland, CA
Doug Schneider, editor of Good Sound, part of the Sound Stage network, wrote this article back in May of 09:

http://www.goodsound.com/editorial/200905.htm

They might be the only audio magazine that supports the use of blind listening tests when reviewing audio products.

He followed up with Part 2 a month later:

http://www.goodsound.com/editorial/200906.htm

Part 2 has a great photo of the Harman room used for blind testing of speakers.

Both articles are, IMO, extremely well written. Schneider seems to be keenly aware, honest and investigative, instead of repeating a specific dogma.

IMO Sound Stage currently has the best loudspeaker measurements (done at NRC) of any audio magazine. If one knows how to interpret those measurements, the subjective reviews come into context
 

kach22i

WBF Founding Member
Apr 21, 2010
1,591
210
1,635
Ann Arbor, Michigan
www.kachadoorian.com
Blind testing refers to the practice of concealing from the reviewer the identity of the product under test, in order to eliminate the bias associated with knowing the product’s make, model, price, appearance, etc.

I wonder how similar this may be to just closing your eyes, I had a recent experience with this.

Link to review/experience:
http://www.whatsbestforum.com/showthread.php?454-JansZen-electrostatic&p=5274#post5274

The last obstacle to point out has to do with the surprising level of resistance still met by even well-thought-out blind tests.

I might be one of those people who fear a blind listening test.

1. I had mean older sisters who once convinced me to play "Blind man's Bluff", a game I played only once. They were evil little things I tell ya.

2. Trusting someone else to set up speakers not to your personal preference often leads to poor results. Someone just set up stats too far apart on me (stats should be closer together than conventional speakers in my opinion) and I think it colored the whole review. See link already given.
 

MylesBAstor

Well-Known Member
Apr 20, 2010
11,236
81
1,725
New York City
DBT DO NOT WORK for audio. Plain and simple. Trying to apply a method to all situations without understanding the test's downsides, limitations and problems is just plain bad science. Just because DBT works for drug testing does not mean it will work for audio!!!

1. Short term memory is notoriously unreliable. Short term memory has a serial processing bottleneck that affects perception. Thus limited "disc" storage space.
2. CNS arousal aka inverse U effect-eg. one wants least arousal and most perception for test. This can't be met.
3. Interindividual hearing differences swamps out statistical tests.
4. DBT is designed for a null effect.
5. No internal controls.
6. Unfamiliarity of equipment, room, sources in most cases.
7. How can you prove adding another piece of gear in the signal path doesn't affect the sound? As John Curl said, every connection affects the sound.
8. Oh, and these individuals are just a little biased? They're hardly unbiased sources. Why don't you find some things from the other side of the argument. MY PhD advisor would label your argument as parochial-only taking the data that supports your point of view and ignoring that data that doesn't support your viewl. Life and science unfortunately is not black and white but shades or gray, something that engineers only find out after they graduate. (at least the good ones).

Need I add more?
 

FrantzM

Member Sponsor & WBF Founding Member
Apr 20, 2010
6,455
29
405
Myles

I am not a huge fan of Double Blind Testing. The procedure is more complicated than it sounds. Takes longer than people realize. and the results are not easy to interpret. Yet rejecting "en masse" blind testing and again proclaiming with no verbal modifier that DBT DO NOT WORK for audio. Plain and simple. is plain and simple inaccurate.
It does work if when you don't know you can't identify reliably. You should be able to identify a component reliably, especially if when, aided with your eyes, you are able to qualify the component. We fail miserably for some things we have come to label components, cables in particular. Repeatability is at the core of all our scientific progresses from the Shellac 78 to the LP or the R2R to the CD and SACD and HRx... It is true that we don't know it all and that several aspects of our knowledge is diffuse at best but repeatability remains central to Scientific process and progress. If we cannot repeatably (reliably) discern a component when visual clues are removed , doesn't that mean something ? That we are aided by our eyes and previous knowledge in our evaluation or qualification?
I have too often seen the notion of short term memory turned on its head to show why BT, not DBT by the way doesn't work in audio... Why are we then able to identify once visual clues are re-established? Also, let us not derail the discussion by mentioning the fact of unknown speaker positions. One can easily device a test where the speakers are positioned correctly and the test conducted blind, i-e NOT knowing what is being played.
I also would like to understand what you mean by DBT is designed for a null result ?
What do you mean by "no Internal controls"?
Your point 7 about John Curl is not ap roof really , rather an anecdoctical reference , the effect would have to be the same for all the equipment under test, and if the effects fool you equally, well the conclusion remains the same ...
That the people under test be average could be true , no problem there. I have seen my niece, a violin player, identifying her violin playing within an orchestra. She can with her eyes closed recognize her violin .. So yes, the average listener is not the best judge. This argument fails however when a trained audiophile cannot reliably recognize his components.

You do need to add more. DBT can work for Audio under certain conditions it can be a good tool as in anything it has its flaws and it is far from being the definitive sliver bullet some have made of it but useful it remains.

Frantz
 

Gregadd

WBF Founding Member
Apr 20, 2010
10,517
1,774
1,850
Metro DC
I would hope we would not be tricked to go down this road. If you wan to take on a mythical industry why pick on us lowly audiophiles. Try taking on the biggest and most successful purveyor of myths the world has ever known. The American advertising industry. They manipulate our fear, ignorance, greed, and insecurity in a matter that has become an art form. Think about it. Our president recieved two awards upon being elected. The Nobel Peace Prize. Because he defeated the war monger republicans. Secondly he received a marketing award for his campaign. Think about it.
Some may say a magazine like Consumer Reports takes on the industry. Really. Consumer reports sells as much junk as Madison Avenue.. All they do is weed out the worst offenders. When consumers pick up that magazine they are usually ready to buy and head out to the store with confidence. What more could a retailer want?

Is there any other industry that subjects their end users to double blind testing? If I want to buy a bass fishing boat with sparkle paint, twin mercury engines, trolling motor, live well, and a fish finder that rivals military radar nobody cares. Does anyone doubt there are great fisherman out there who use tactics that have no scientific basis? In the end audiophiles are not scientist we are hobbyists. Tweaking is fun. Maybe putting dumbbells on top my transformers does not affect the sound. I don't use them to exercise anymore. It makes them more difficult to knock over when I've had to much to drink.

I am lawyer. That means I argue for a living. I can only marvel at the the beauty of the ABX/DBT debate. It is pure geneious. I wish I could use it. You never have to prove anything. The proponent of ABX/DBT does not even have to have any audiophile credentials. E.G. years of listening to all kinds of audio equipment under different conditions. Even if your opponent makes the right choice he has to choose again. Even if he makes a right choice it is statistically insignificant. No one requires pharmaceutical trials to cure the patient of a disease ten times before it is statistically significant. There is a certain brilliance to this "Heads I win Tails you lose" philosophy. It happens all the time now. If you want to defeat a well accepted government policy. Don't argue against the policy. Concede its' good policy. Just argue that the government has no power to enforce it.
With ABX/DBT you don't have to prove vinyl is better than digital or vice versa. Just argue that those who claim vinyl is better utilized a flawed evaluation procedure. Hell, you don't even have to prove anything yourself or offer a better test. You just exploit the confusion. Magnificent.
 

es347

VIP/Donor & WBF Founding Member
Apr 20, 2010
1,577
35
970
Midwest fly over state..
Life and science unfortunately is not black and white but shades or gray, something that engineers only find out after they graduate. (at least the good ones).

Really? Seems a little broad brush Myles. Shame on ya.
 

MylesBAstor

Well-Known Member
Apr 20, 2010
11,236
81
1,725
New York City
Not from my experience with undergrad engineers. After grad, they learn that what's in the book isn't quite true and black and white. Hey that's true for science as a whole.
 

FrantzM

Member Sponsor & WBF Founding Member
Apr 20, 2010
6,455
29
405
Hi

I am not sure I see the point here completely . I am by no means saying atht an audiophile is a scientist. No. He relies on his auditory system to make decisions. He/she hears and decides if what he's heard is acceptable.
It remains however that it is helpful for the hobbyist to test the objectivity of some of his/her observations. That you put a header on your car because you like the sound is one thing that you claim and maintain that the header makes it go faster must be sustained by tests and repeatable, else .... Same with the example about the fisherman and his magic bait. It must most of the time catch the fish, else ...
In our endeavor the quest for better sound, we have Science and have made pretty good use of it. Several aspects of that science are not entirely well understood, even less measured, they remain repeatable. That is what good components do. Their inclusion in the reproductive chain repeatably produces good results for most people anyway. Equipment such as preamps , amplifiers, DAC and speakers are clear to discern and repeatably so ... It is good once in a while to see how we would have reacted if the gear wasn't known .. This is a Blind test.. nothing "scientific" just not knowing ... what is .. even in our own system and honestly describing our observations. It helps us deriving more enjoyment from our system I would think ... Make it better in a reliable way...
I am for example extremely interested in Audio reviews. When I was in Haiti , that was the only way to remain connected with the s SOTA, since there is no High End Store in my neck of the hood. The truly useful reviewers are those that are capable of recognizing their biases and take measure to alleviate them in their reviews. and there, blind testing helps. It helps a person compare honestly a lowly amp from a an unknown with one with greater renown and pedigree. It helps me as an audiophile not immediately connected to make a better decision and even if I were connected removing the visual clues or some knowledge of the components help me make a better decision. That is to repat myself one last time what blind testing sometimes brings to the table.

Again, some may enjoy the journey. Oh I know I did! Changing equipment every other year even other 6 months. Resulting in a lot of side moves and few if any progress. I wasn't listening to music, I was running after components .. I have in the meantime, realized the value of proper set-up especially that of the room speaker-interface and am now more concerned to extract the maximum re from my equipment and am willing for that to use all the tools available.. Blind Testing , in some cases is one of these tools and a good one at that.


Frantz
 

rsbeck

WBF Founding Member
Apr 20, 2010
848
11
0
DBT is the only way to prove that there is a difference to hear. In the absence this kind of verification, there is no proof. This is just a plain hard scientific fact.

It has also been proven many times over that your brain can fool you into hearing differences that do not exist in reality.

The audiophile world is rife with anecdotal claims that strain the bounds of credulity.

It is also stocked with self proclaimed authorities who base their authority on their claimed but untested auditory superiority.

It doesn't take much to figure out why some of these authorities would resist a method of verification that can expose a fraud.

If we wish to be honest, we will all admit that there are ridiculous anecdotal testimonials attached to all kinds of things in the audiophile world.

You can regularly find people on audiophile sites who will claim the ability to hear the un-hearable.

You can find testimonials from college kids with a $250 DVD player for a front end and $500 speakers who will tell you that upgrading to $1,000 speaker cables enabled them to hear detail they never heard before, that their soundstage grew to an incredible size and depth, etc. etc. IMO, it becomes rather sad. If we want to help that kid, IMO, we should help him understand that his money would have achieved for him much greater benefits elsewhere in his chain. But, who is going to set that kid straight?

Before we're tempted to chuckle at that kid's testimonial, let's admit that this type of anecdotal evidence is trotted out regularly at all price points.

Now, when someone has claimed that some cable or other piece of gear removed the veil, enabled the listener to hear things never heard before, that the sound stage expanded to immense proportions ---- and then that listener cannot even reliably identify this cable or other piece of gear in a DB A/B test, it's time to call B.S.

DBT absolutely works, but you have to understand what DBT does -- it proves beyond the shadow of a doubt that an difference exists in reality.

Without that proof, any claim is unsubstantiated.

That's a hard scientific fact.

So, we travel in a world of mostly unsubstantiated claims and we have to do our best to sniff out the B.S. because if we're honest with ourselves, we know the audio landscape is littered with B.S. land mines and unless you want to be duped into an unproductive and sometimes very costly side-track, your first priority is to hone your B.S. detector.

Having said that, IMO, all this means is that us audiophiles ought to have a little more scientific humility.

IMO, there's nothing wrong with saying, "I think I hear X, but I have not verified it and I could be imagining it."

Or, "to my ears, I think the difference was __________."

Or, "if you can't hear what I hear, it could be that I am imagining it."

Or, "but I didn't level match the components, so there could be other explanations for the differences I think I heard."

Or, "but I doubt I could pass a DBT if tested on this."

All this would do is remove a lot of fake authoritativeness and turn this into what it should be, IMO, a sharing of admittedly flawed and subjective experience between a group of people who are trying to muddle through with unreliable tools which include our ears and brain.

Because if audio memory is too short for an a/b test, it is obviously also too short for any test and every audition is therefore flawed.

Having said all of that, I also have to say that I hate when DBT's are introduced into most discussions because proper DBT with peer reviewed results are just not obtainable in most instances so all it does is short circuit a discussion.

Forget about DBT's and lose a lot of the fake authoritativeness and IMO we'd have better discussions.

Hopefully, right here.
 

MylesBAstor

Well-Known Member
Apr 20, 2010
11,236
81
1,725
New York City
You have failed to address one single point that I have made regarding the issues with DBT. To accept any methodology blindly without question is about as unscientific as it comes. For instance, when I review an article for a journal, I go through the materials and methods section to ensure that the the assays done are appropriate for the hypothesis being tested, proper measurements were done, etc. If the proper internal controls haven't been done, then the measurements are meaningless. And with every scientific experiment, there are certain assumptions that have to be made since no methodology is without unknown or known variables. If you don't accept the author's assumptions, then this paper is of little use to one.

Number two, who cares how people spend their money? Do you tell them that its a rip off to buy a Ferrari when you can get a Kia that also does 60 mph? Or a Patek since a Casio keeps better time? Is there some sort of audio police that I don't know about?
 

es347

VIP/Donor & WBF Founding Member
Apr 20, 2010
1,577
35
970
Midwest fly over state..
Not from my experience with undergrad engineers. After grad, they learn that what's in the book isn't quite true and black and white. Hey that's true for science as a whole.

Just for grins, what is your background, both educational and experiential?
 

Steve Williams

Site Founder, Site Owner, Administrator
Myles

Number two, who cares how people spend their money? Do you tell them that its a rip off to buy a Ferrari when you can get a Kia that also does 60 mph? Or a Patek since a Casio keeps better time? Is there some sort of audio police that I don't know about?

Your post exemplifies what for me has always been my mantra in this hobby......."let your ears and wallets be your guide because after all it is your ass in the sweet spot"
 

Gregadd

WBF Founding Member
Apr 20, 2010
10,517
1,774
1,850
Metro DC
Here we go sinking down into those endless arguments that change the opinion of no one. You tell me. Beleive me no one has. If 99% of ABX/DBT test results reveal no statistical significance between components then what is the point? That is, we have shown that no one being tested was able to discern an audible difference between the equipment under test. Lets' assume we have designed the component as best we can under known parameters and measurable scientific tests. We can all concede it still does not sound like real music. The only remaining purpose for the test is to discredit those who claim to hear a difference.

If the DBT/ABX is a true scientific tool then use it to make a better product. We can of course make perfect equipment based on our known parameters and our ability to measure. Like the scientist who tried to do a DBT with steroids failed because the difference was so great between the steroids and the placebo they could not decieve anyone. If you know all the parameters that matter and can measure them, you should be able to design a system where they are vanishingly low. Then why does it sound nothing at all like real music. I have a low distortion speaker. The ML CLS. It is as about as transparent as you can get.
All you ABX/ DBT proponents put that ABX/DBT test together and I will take it tomorrow. We know real music will win. To make it fair we could make a recording and have the same group play at the test. The point is that something was lost in theat playback process we can't measure. In fact there is a huge gap. If we are to bridge that gap we are going to do it by trial and error. Until we come up with some better measurement techniques. Some advances are real some are snake oil.
 

MylesBAstor

Well-Known Member
Apr 20, 2010
11,236
81
1,725
New York City
OK, just for grins. BA Biology, PhD Radiation Biology and Pathobiology, Columbia University. 20 years of academics and cancer research dealing with free radical chemistry, cellular redox potential, the biochemistry of chemotherapy drugs and/or their interaction with ionizing radiation at Columbia College of P&S/Columbia Univ and New York Medical College. At NYMC, I won a prestigous grant from The Brain Tumor Society for glioma research. I have 30+ scientific, peer reviewed publications in such journals as Cancer Research, Radiation Research, etc. In addition, reviewed submitted research for a number of scientific journals. Plus won several awards for my research including the Hoffman La Roche clinical award for my PhD dissertation and an International Congress for Cancer Research fellowship to study in Paris at the Institut Gustav Roussy, plus some junior investigator awards. So I think I'm well versed in scientific research.

In the end, I left as an Associate Professor to start my own high-end audio magazine for many reasons.
 

rsbeck

WBF Founding Member
Apr 20, 2010
848
11
0
To accept any methodology blindly without question is about as unscientific as it comes.

Accepting any kind of authority without question is rather silly, IMO, but it is just a plain hard scientific fact that in absence of verification, any claim is unsubstantiated.

Do you tell them that its a rip off to buy a Ferrari when you can get a Kia that also does 60 mph? Or a Patek since a Casio keeps better time? Is there some sort of audio police that I don't know about?

LOL. For that matter, who cares about anything. Caring is silly, right? A Patek looks better than a Casio. There's no BS claim that if you can experience the superior time keeping ability of the Patek, you possess amazing time experiencing faculties. And we know that anyone trying to claim such a thing would be full of dirigible fuel. If someone wants to achieve quality time keeping and we want to give responsible advice, that advice would have more integrity if it were honest rather than laced with fake authoritative B.S. The real answer would be that the less expensive Casio will do just as good or possibly better job of keep time, so if that's the goal the Casio will do the job. If something else is desired, status or appearance, then that's what the Patek offers. And then, I agree, if someone wants to spend the money on a Ferarri, a Patek, a Yacht, a 100,000 square foot house, who cares?
 

audioguy

WBF Founding Member
Apr 20, 2010
2,794
73
1,635
Near Atlanta, GA but not too near!
I am always amazed when this subject comes up as the conclusion is always the same: No one's mind is changed!! A significant issue with this hobby is that HUGE amounts of "fraudulent" product is marketed with clearly inaccurate/misleading/plain wrong "scientific data" and folks buy those products on a regular basis.

Can't think of any other hobby where the camps are so widely divided on subjects like blind testing, cable differences, etc. In wine tasting, for example, if someone were to say "I prefer red wine" I would be surprised if another would angrily respond "you are wrong as white wine is clearly superior!!!!!".

I am also aware of many who have indulged in blind tasting for wine differences!!

Oh well!!!

While I do think blind testing is a valuable tool in dismissing some of these fake products, if you have the money and want to spend it on something (scientific rip off or otherwise) have fun. It should not be, but clearly "buyer beware" must be the watchword !!

And I have used blind (not double blind) testing to clearly prove to me that what I thought I heard I did not hear. When I knew what I was listening to, I could always hear the difference. When I did NOT know what I was listening to, I could not. Pretty simple to me. If there were any differences that were there that I could not reliably and repeatedly identify, they clearly could not have been very important to me. For me, blind testing has save me a lot of money. YMMV
 

FrantzM

Member Sponsor & WBF Founding Member
Apr 20, 2010
6,455
29
405
Accepting any kind of authority without question is rather silly, IMO, but it is just a plain hard scientific fact that in absence of verification, any claim is unsubstantiated.



LOL. For that matter, who cares about anything. Caring is silly, right? A Patek looks better than a Casio. There's no BS claim that if you can experience the superior time keeping ability of the Patek, you possess amazing time experiencing faculties. And we know that anyone trying to claim such a thing would be full of dirigible fuel. If someone wants to achieve quality time keeping and we want to give responsible advice, that advice would have more integrity if it were honest rather than laced with fake authoritative B.S. The real answer would be that the less expensive Casio will do just as good or possibly better job of keep time, so if that's the goal the Casio will do the job. If something else is desired, status or appearance, then that's what the Patek offers. And then, I agree, if someone wants to spend the money on a Ferarri, a Patek, a Yacht, a 100,000 square foot house, who cares?

I also don't think the comparisons arer valid... The Patek cannot claim any superior Time keeping ability over the Casio.. easily provable... and their owners , for the most part I hope would not dare claim it better at that task than any Quartz watch... We are already agreeing that the watch is a piece of jewelry ... Could it it be the case in Audio? We know not, we hope not...
A superior component in Audio must do something superior beyond just being more expensive . Let's not revert to the flat-earth rationale... As some call it: Audio Relativism... There is value in knowing why a gear is better and this must be reliable.. it must be repeatable.

I don't think any advance can be made through adherence through dogma and faulty reasoning to try to prove a point. DBT is usually not feasible. It is not a solve-all... yet in some way it illuminates that certain point of view to which we, audiophiles have adhered for a long time, are at least suspect... and I am being nice ...

Frantz
 

audioguy

WBF Founding Member
Apr 20, 2010
2,794
73
1,635
Near Atlanta, GA but not too near!
It is not a solve-all... yet in some way it illuminates that certain point of view to which we, audiophiles have adhered for a long time, are at least suspect... and I am being nice ...Frantz

It may not be a "solve ALL" but I still believe it has a place to debunk some clearly fraudulent claims. And yes, you are being very, very nice --- as usual !!
 

es347

VIP/Donor & WBF Founding Member
Apr 20, 2010
1,577
35
970
Midwest fly over state..
OK, just for grins. BA Biology, PhD Radiation Biology and Pathobiology, Columbia University. 20 years of academics and cancer research dealing with free radical chemistry, cellular redox potential, the biochemistry of chemotherapy drugs and/or their interaction with ionizing radiation at Columbia College of P&S/Columbia Univ and New York Medical College. At NYMC, I won a prestigous grant from The Brain Tumor Society for glioma research. I have 30+ scientific, peer reviewed publications in such journals as Cancer Research, Radiation Research, etc. In addition, reviewed submitted research for a number of scientific journals. Plus won several awards for my research including the Hoffman La Roche clinical award for my PhD dissertation and an International Congress for Cancer Research fellowship to study in Paris at the Institut Gustav Roussy, plus some junior investigator awards. So I think I'm well versed in scientific research.

In the end, I left as an Associate Professor to start my own high-end audio magazine for many reasons.

With a name like Myles Astor, I should have known better to ask such a question. If your ring was within kissing distance, I would :)
 

astrotoy

VIP/Donor
May 24, 2010
1,547
1,017
1,715
SF Bay Area
I have a slightly different take on DBT. While it may be useful, it isn't scientific or objective. There is an implied comparison with scientific, typically medical, research, DBT. In those tests, rarely if ever is the patient asked what s/he feels about the treatment. Do you think the tumor is getting smaller? There is an independent, objective measurement made - like an xray of the tumor before and after treatment, compared to a large number of patients who have received or not received the treatment. On those rare occasions that they are asked their opinion, it is with a very specific rubric - how long is your migraine, what is the frequency, what is the maximum intensity of pain from 1 to 10. What a true audiophile DBT would do is eliminate the subjective part of the test - the listener's opinion - and connect the listener's body to devices which would measure changes in hormonal levels and brain activity which would indicate that the person is or is not experience real changes in emotional or other responses and whether those changes are positive or negative - e.g. increases in endorphin levels.

Further, I would argue that if a listener is trained to listen for specific things in a recording, they can easily hear differences that an untrained listener would not hear. People don't have to born with golden ears, but they need to be trained. I had a very interesting experience with a trained listener, my wife. We have listened together to a large number of the recordings of the Beethoven piano concerti. My wife is a very good amateur pianist who has played all of the Beethoven piano concerti with orchestras. One day we were listening to one of the Rubinstein performances of the Beethoven PC5, which I had heard quite a few times. At one point she said - "He's cheating - he's rolling some octaves when he should be playing them in unison. I know that, because that is what I do. It's a pretty difficult passage otherwise." Well, we played it again and I could now clearly hear what she was saying. And furthermore, in any further recording we have played, I will hear immediately whether the pianist is cheating or not. However, without that little lesson, I most certainly would never have heard the difference between performances. It isn't subtle, but there are so many things going on, the typical listener won't hear even gross differences.

Larry (BTW, for her day job, my wife has been a research professor at UCSF for the past 30 years and primarily does NIH funded research - over $20M as principal investigator, including lots of DB studies.)
 

About us

  • What’s Best Forum is THE forum for high end audio, product reviews, advice and sharing experiences on the best of everything else. This is THE place where audiophiles and audio companies discuss vintage, contemporary and new audio products, music servers, music streamers, computer audio, digital-to-analog converters, turntables, phono stages, cartridges, reel-to-reel tape machines, speakers, headphones and tube and solid-state amplification. Founded in 2010 What’s Best Forum invites intelligent and courteous people of all interests and backgrounds to describe and discuss the best of everything. From beginners to life-long hobbyists to industry professionals, we enjoy learning about new things and meeting new people, and participating in spirited debates.

Quick Navigation

User Menu

Steve Williams
Site Founder | Site Owner | Administrator
Ron Resnick
Site Co-Owner | Administrator
Julian (The Fixer)
Website Build | Marketing Managersing