I could not find what caused the "stopping recording..." issue, but here is what I advise on the encoding overload.
Try Using Game capture instead of Display capture.
Make sure you are on the latest NVIDIA driver.
To ensure that OBS Studio has the hardware resources it needs for realtime streaming and recording, it is recommend to disable the "Game DVR Background Recording" feature via these instructions:
https://obsproject.com/wiki/How-to-disable-Windows-10-Gaming-Features#game-dvrcaptures
Recording format: mp4 (this is not recommended, but it is to test if your format is the issue. You can change it back if it still is not working or it was something else causing the issue.)
Audio tracks: 1
Encoder: NVIDIA NVENC H.264 (new)
Rescale output: unchecked
Rate Control: CQP
CQ level: 16-20 (start at 16 and raise it by 1 if you are getting encoding overload)
Keyframe interval: 0
Preset: Max Performance (there is no difference in quality between Max Quality and Max Performance as CQP is a quality target not a bitrate target. This may cause larger file sizes, if it is to big try raising it to Quality)
Profile: High
Look ahead: off
Psycho Visual Tuning: off (this can cause you final render to look weird if you are doing Smart resample/frame blending, and take more gpu usage)
GPU: 0
Max B-Frame: 0 (they are unneeded for local recordings and just take more encoding power)
In video set downscale filter to bilinear as you are not downscaling.