Been there, done that, at least with Windows / JRiver. Messy, to say the least. JRiver is very cryptic, plus there are all the video card driver settings, and that's before you start fussing with MADVR! I thought it turned out ok, but was overly complex and unsupportable and now use an Oppo
Recommendation; just save your time and buy something designed to do the job i.e. kaleidescape, and I'd wait until they release their UHD model next year.
Nyal - I suspect that you are many JR releases behind. I started on JR 18, then 19 and now 20. I have never ever had to fiddle with MadVR or even know it existed. I just check the RedOctober HQ option in JR and it handles all the interfacing to MadVR behind the scenes transparently with default settings. I have never had to touch MadVR directly. And, it looks great.
Yes, learning JR is a challenge up front. But, I have found no better tool for my audio library, and I prefer to stick with it for video.
It was you awhile back who strongly advocated for JR in your HTPC thread somewhere. That inspired me to follow that path, which I have done successfully. My system is now prepro-less, TV cable box-less, and separate player-less, with JR as the player and control center. I knew it would be less convenient than having more boxes in my system, but it works well at very high quality, with the occasional hiccup, of course. Windows and software issues, aargh. I do not know of a better, more comprehensive tool to attempt this with, but if it exists, I sure would like to know. But, JR could always be better, and it keeps adding features, which, of course, make it even more complex.