I think the problem is that the remastering engineers think they have better equipment today and feel compelled to muck with the recording

After all, they have to justify buying all those expensive toys. And the latest is autotune
I think the problem is sheer commercial interest exacerbated by fashion. The loudness wars began as an attempt to make recordings cut through the noisy environment in which most people listen to their iPods through open ear buds. Record companies decided they liked it, that louder would get more attention, draw more interest, and it escalated into the fashionable thing to do, then the thing that all were compelled to do to keep from falling into the background of the blare. Still, an awful lot of music has barely been affected, and if you don't listen to a lot of commercial pop, you won't have to put up with it much. The Beatles remasters are not brick-walled, by the way. They don't have a ton of dynamic range, but they never did.
The loudness wars are not new, by the way. This is just the latest battle. There was plenty of compression of singles, so they would cut through the clutter on the radio and be heard loud, if not clear, on car radios with the widows rolled down, going all the way back to the 50s (Phil Spector's classic records were very compressed). Then, as now, it was mostly limited to pop. Jazz and classical never suffered from the syndrome. It's so popular now, though, that it has seeped into some less popular forms, but thankfully in a very toned-down version. You won't find many truly brick-walled recordings in Bluegrass, Americana, Folk, Traditional Country, Jazz, Blues, etc. The last couple of "new" artists I discovered -- Ray LaMontagne and Madeline Peroux, have some beautifully produced and mastered records. They are a bit louder than pre-"loudness wars" era recordings, but not enough to crush the dynamics, and they suffer from none of the treble-juiced harshness that is so common on pop records. Because this wave of bad mastering is focused mostly on pop records, I remain personally and happily unaffected. Seldom has painfully unhip been so lucky.
Tim