USB as a data transfer for high end audio was never designed for that purpose.
I've heard this slogan on more than one forum I think but it really doesn't make any sense. Data is data, and USB async most certainly was designed to get the best audio performance (in theory) possible by allowing the DAC to be clock master. In this sense USB async easily beats AES where the transmitter is the clock master and the DAC has to slave itself to that.
In my estimation the problems with USB don't so much arise because USB itself isn't 'fit for purpose' but because computers and their switching supplies are incredibly noisy beasties and USB does not provide isolation as standard.
That and the fact noise and contamination propagates the data stream.
Right on, not just the data stream though but also the cable screen and the power supply wires.
The amount of negative impact will depend on how much investment and good design has been implemented in the USB output on the server and input at the DAC.
Not just the degree of isolation provided by the USB output but also the amount of noise being generated by the computer and its power supply. All else being equal a lower powered computer is going to be less noisy than a higher powered one. Which is why the newer generation of ARM-based devices are quite good news for audio as they tend to be less power hungry than the older generation Intel-based ones.