Yes, of course we hear it all together but in looking for ways to correlate what we hear with measurement it is somewhat cleaner than IM harmonics + the other distortions you have mentioned, which may be very difficult to actually measure. Pass did a white paper a few years ago where he showed how IM distortion can build up in electronics and he called it the "elephant in the room". Crowhurst likened the effects to a signal modulated noise floor.
By definition:
Given a system, if in the spectrum of the response there are spectral components that were not present in the stimulus, the system is said to be
nonlinear and the extra components are classified as
nonlinear distortion (regardless of the cause).
Mathematically, the transfer function of a nonlinear system also depends on the amplitude and shape of the stimulus (hence the impulse function is not h(t) but h(t, s(t)).
Distortion is not difficult to measure: it is sufficient to compare spectra (which can be done with great accuracy and precision).
As far as a sound reproduction system is concerned, distortion can be audible or not audible. Audible distortion can be tolerated or not tolerated.
If the distortion is not audible, it is as if it was not there; if it is not tolerable, the system can be thrown away.
The problem of audible and tolerable distortion remains. This one exists, it can be heard, but it hurts only a little. This is largely subjective (some, for example, like it).
Distortion adds information that was not present in the stimulus: this "bad" extra information can mask the "good" one. So the effect of distortion is always masking. Even noise has a masking effect, but less severe than distortion because it is not related to the stimulus and does not translate more information but only "disturbance". Noise contributes to Listening Fatigue.
The solution is to point directly to systems that produce
non audible distortion in actual conditions of use, and the most important thing to define is the perception threshold or JDD (depending on the type of measurement made). The most significant (though not "ideal") measurement for a loudspeaker is the measurement of Integral Distortion with multitone signals (the JDD is 0.32% at 90 dB so the spectrum of the distortion must be at least 50 dB below the spectrum of the stimulus). For electronics the significant measurements are Null Test and Random Distortion (which have different JDD).
A playback chain is made up of source, amplifier and speakers. All the distortion produced by the source goes into the amplifier and that produced by the amplifier goes to the speakers which distort on their own. At each step the distortion, if any, increases and an non audible distortion in the source can become audible. It follows that if a 0.32% distortion for a loudspeaker is non audible, that one of the amplifier will have to be 0.1% and that one of the source at 0.032% (this is just an example to understand). Less is better. So we should not read the distortion data all in the same way.
Then there are tube amplifiers and their particular relationship with aural distortion, but this is all another story because it is strongly related to subjectivity. I will say more: a chain formed by analogue turntable, tube amplifier and horn loudspeakers or "mono broad band" is not judged instrumentally: you either like it or dislike it.
This is the situation. Then many things can be said, but in substance they have to agree with the above. Audible but tolerable distortion has a limit: sooner or later you find a musical program that makes you hear it. This is solved with information: those who buy a pair of mini-speakers must be aware that they are suitable for certain things and not for others.