I think the obsessed audiophile longs for the simple experience he remembers from days past when he listened to music at a dorm party or with a girl in his car and did not think about the sound so much.
I agree completely with you here.
I have a feeling this also comes with age. As we age, our hearing deteriorates, perhaps we seek a different kind of sound.