Run the Auto-configuration Wizard (tools menu) and apply the settings it gives you. Restart OBS and test.
This are the best reliable settings for your hardware.
The GPU isn't NVENC supported (check link bellow) so you have to use x264
Find the related video encoding and decoding support for all NVIDIA GPU products.
developer.nvidia.com