Alright, I've talked it over with Elgato, and we're going for a drastic approach here to identify if the issue is something with OBS, or your system.
They confirmed that their software uses the same DirectShow interface that is exposed to OBS, so there HAS to be something going on there. Can you grab the trial version of xSplit and test the capture card there if you have the same kind of performance issues?
Sorry for taking so long to get back, it's been a few days of back and forth with Elgato and myself.