Exactly my take on the matter as well. Indeed, while my anecdotal conclusion is that the higher res file sounds better, in most cases the lack of an absolute leads me to assume the differences are in the pre-production stages: mastering, etc. In many instances I wonder if more care is taken when mastering the higher res sample(!).
This is it, in fact. On occasion when comparing format samples I have often asked myself what am I really comparing? The CD vs the 24/96 files, the mastering thereof, the CD transport vs the streamer...
At the end of the day, being disinclined to reach an absolute conclusion, I just listen to what sounds best to me - usually the 24/96 - 192 - dsd - or, (best of all) DXD...