The damping factor is a result of a division of load impedance and amplifiers' output impedance. We want our amplifier to be a perfect voltage source, which means it has zero output impedance. It means that output voltage stays the same despite differences in load (eg. different speakers). What zero impedance means is that with a certain voltage output amplifier can deliver infinite current. Well, that is never the case as we all know. Usually, amplifiers' output stage limits the current capability more than the output impedance.
It's different with tube amplifiers that have naturally higher output impedance. That is a factor why most tube amps prefer stable impedance in speakers. They have trouble with big drops of speakers' impedance because their response is "slowed" by their own output impedance. To make it a little clearer: if you plug speakers rated 4Ohms to 4Ohms tap in a tube amp you want the speaker to be 4Ohm regardless of the frequency. Some speakers are like that, some aren't.