A GTX 1060 won't give you the best quality with NVENC, I currently use a 1070 and like your card it struggles to achieve quality even at higher encoder settings, it's a limitation of the Pascal NVENC implementation. I ended up using x264 on my i9-9900K.
If you have a fast CPU, x264 encoding will give you better results for a given bit rate. Either way, if you are encoding a 1080p60 stream or even a 720p60 stream at only 3 or 4 Mbit/sec your quality will suffer, you need to give NVENC more bit rate to get the same visual quality as x264. For offline recording for later use, you can increase the bit rate and it's not so important, but NVENC on the 10 series cards isn't as efficient as CPU x264.
To achieve highest quality many streamers use a second machine just for encoding, either capturing HDMI output from the game PC or sending full-screen NDI. That way you can do more intensive things like a two-pass CPU x264 encode without compromising game quality, plus do a high bit rate recording for later use, and you avoid all of the problems with single-machine display capture & encoding.
YouTubers will be using OBS to capture gameplay but recording at a far higher quality and bit rate for edits. For example, using FFmpeg to save ProRes HQ 4:2:2 at 100 MBit/sec for editing and uploading will result in a much better quality final version, after YouTube performs its own transcoding on your upload.
Provided you have adequate disk space and a fast enough disk, a recommended minimum quality for capturing footage for editing/YouTube publishing would be something like at least 75 MBit/sec (or for H.264, something
very high, like x264 CRF 8 / NVENC CQP 6, or a lossless mode).
Related watching and reading:
Enjoy the videos and music you love, upload original content, and share it all with friends, family, and the world on YouTube.
www.youtube.com
StreamFX incorporates more FFmpeg-based encoder options for OBS (which were previously
in their own plugin).
To avoid potential data loss, I record recording MKV and use the File -> Remux Recordings feature to remux to MP4 if needed. I can also use MKVToolnix software to manipulate or extract the MKV audio and video streams as needed.
ProRes or ProRes HQ does require a more powerful machine to encode and edit, however its picture quality preserves much more detail than H.264 unless you record lossless (intra). However, with the right settings, you can record NVENC lossless with minimal CPU impact and using a lot less disk space.
What is it you want to achieve, the highest quality possible for streaming, or a setup dedicated to recording footage for editing and publishing to YouTube?
