That is the same argument that the solid state measurement people like to make.Only above a certain level…below that level it is inaudible.
It does not work that way for them and does not work that way for you or anyone else. Distortion is a modifier. It is not a separate signal.
It does indeed color the sound, less so at lower volumes, and its always there.
Put another way, the distortion of any amplifier is its 'sonic signature' plain and simple. For that reason its best to keep whatever distortion as benign as possible. The only really 'benign' distortion is the 2nd and 3rd.