Did you try the "Edit Scene" button to properly shape/position the Game Capture source? There's really no reason whatsoever to go through the difficulty of setting up Dxtory when OBS already does the same thing.
Chances are you accidentally nudged the scene out of place or, for whatever reason, it defaulted to that position so you just need to nudge it back. There's also the "Stretch image to screen" option which may help? Not really sure.. never needed to try it haha