For recording you should use CQP instead of CBR. Set the CQ level between 23 and 15 (lower number bettwr quality), keyframe 2 and B-frames 4.
Check for driver updates too.
An strange thing is in the images you have set canvas to 2560x1440 but in the log file is 1080p canvas and output.
The canvas is the base resolution of your monitor. If you are using a 1080p monitor, set that. If you have a 2k monitor, set the 2k value,same as it's 4k. Then you can set the output resolution to a different resolution scaling it. Best result is using Lanczos (36).
Display capture is looking into your monitor, so if it don't show your monitor the issue most probably it's the driver. If you have 2 monitors, make sure you're selecting the correct one.
As the issue continues with game-capture, I tend to think it's the driver.
Some times you're forced to do a clean reinstall of the drivers which it's a pain but fixes most issues. There's a guide on how to do it, just search for it.
Hope any of this helps.
