Nvenc has awesome potential in terms of quality, but for streaming in HD on 60FPS it is a nightmare. CPU encoding offers the best quality versus stability and compatability for the viewers on the stream.
When recording and taking note of the filesize, I think it is really compact and not any different from the filesize on CPU encoding, but when I watch back a stream on twitch and have the taskbar open the CPU encoded stream downloads continous a few Mbps and the nvenc stream goes up and down from 5 seconds on 0mbps to 30~60mbps for 5 seconds. Also fo watching back the stream on twitch the CPU on my tablet goes mad. Hitting 80% continously for the nvenc stream while the CPU encoded stream only stresses the cpu on my tablet for 20 ~ 30%.
@OP
If you want semi stable nvenc settings for streaming. Tick on advanced options, QP 1 up to 35 ~ 40, You can experiment with lower values which improve the quality of the image, but artifacts and continous buffering occurs on the stream. Also OBSM tends to crash and it will stop updating frames so it keeps sending the same frame to the stream.