Hey mate, in your recording tab, set the "type" drop-down to "standard", recording format MKV, choose x264 as your encoder. CPU Usage Preset make it "UltraFast". In the "x264 options" field, put this "qp=0".
The rest of the options in the tab don't matter.
This will record your footage in lossless quality. The file size will be HUGE. But you will get a beautiful video and it will impact your game very little. If this causes your frame rate to drop, I don't know how you can fix it. You might be able to use AMD AMF, but I don't know how sorry, I don't have one of those graphics cards to check.