First
CBR for streaming
CQP or CRF for recording
if you record with CBR it doesn't matter what color range you use it will always turnout bad.
Beside that most encoders don't have a efficient way to record RGB and will drop frames.
I don't know about nVidia, but this is what is possible using and AMD card (RX 550 ~ GT 1030)
The footage recorded with these settings has amazing quality and thanks to RGB/Full all the colours are accurate as well. (the only "bad" thing being the AAC audio codec even with a 320 bitrate, and I have no idea how to set exactly all of this up using the "ffmpeg" option which allows different audio codecs)
The only thing that needs to be adjusted from game to game are the I-Frame and P-Frame numbers, games with lots of details (foliage, etc.) like in Fallout 4/3 will require it to go up to 28 to not have the encoder overload, which will result in slightly worse image quality where there is lots of details.
The final file size is a bit larger but who cares about that, when recording footage to edit and then upload to YouTube you'd want it to have near lossless quality anyway as well as the final rendered video will be much smaller anyway.
And judging by how many people praise nVidias encoder, I'm certian that it can do exactly the same.
I just don't happen to own a nVidia card to help with properly setting it up to record gameplay footage.