Hi Caesar,
This is my opinion and largely unsubstantiated; nevertheless, it is logarithmically harder to produce good sounding, high powered amps. In other words, twice the power will be four times more difficult to produce, four times the power, sixteen times harder, and so on. First, the signal has to run through a lot more "stuff" before it reaches the speaker and second, often with higher power comes output devices parallel and they must be matched/QC. That's the problem with good sounding, inefficient speakers: you've got to find a good sounding, high powered amp which is extremely rare and very expensive.
Also, amps typically have a linear range so to speak. At idle and full power, amps are not so good. With a flea watt amp, you're quickly off idle and into the low distortion linear range. With high powered amps, they may have to make several watts to be in their linear range (including an ideal temperature).
For your 97 dB/W/m example, if your are listening at an average of 85 dB, your amp need to supply 0.063W! If your cranking along at 95 dB, you'd need 0.630W. Thus the "first watt" school of thought/
Qualifier: my expertise is loudspeaker and thier motors: I am not electronics engineer. But, I am eager to learn should someone post a correction to my unproven theories.