Switch to CQP/CRF instead of VBR or CBR.
It's NOT advised to record to RGBA in most cases.
NEVER EVER record directly to MP4 in OBS, for ANY reason. Record to MKV if you need multitrack audio, or FLV if you don't. If you need MP4s for editing later, you can go to the File menu, Remux Recordings to convert them over very quickly.
Really, you should create a new Profile (to get default settings), leave the Output in Simple mode, select the Indistinguishable quality option, NVENC (new) as the encoder. It will be visually lossless.
It kinda looks like you followed a horrible 'best settings guide' from somewhere, honestly. A lot of the settings you're using are going to cause problems.