C3 Your literally giving this guy bad information man?!
I was compelled to create an account and correct this guy's mistruth, because it looks like your under the impression that you know what your talking about.
Kid wants good quality, and he is playing a high CPU intensive game like ARMA 3, and asking you for help on a good preset.
And you tell him to NOT use his GTX 1080 which will only get about 40% Usage on a game like Arma 3, you then tell him that NEVER use NVENC because in your uneducated opinion you think it looks bad on all settings.
Clearly you do not know what you are talking about.
Im writing this to correct this guy, and to properly help anyone that has come here with the same question Schannikk had.
NVENC is not a bad encoder, infact it is VERY good in some circumstances, for example Schannikk would see a HUGE difference when using it on Arma 3.
Arma 3 is a VERY CPU intensive game, it also is optimized and likes to put 80% of your games load on your first 3 cores, meaning regardless of what CPU you have, this game will chug in certain scenes. to prevent that the dev's have attempting to create multicore support, by having a few categories of things rendered on your other CPU cores hence the 20%
WIth that said, your cpu is literally NEEDS to have arma 3 all to itself in order to maintain a proper frame rate in deathly full servers like an Exile one.
Trying to stream on top of that will REALLY kill performance on busy servers, and that gets me to my 2nd part, Arma does not use the GPU like it uses the CPU, infact it barely goes over 50% usage for me personally on a GTX 760, and only does so when im doing a small mission and doing 80fps..
So that obviously means NVENC time because the CPU is all booked up and your GPU is sitting on its ass...
That said, X264 gives perfect quality and is statistically better then NVENC. But on Arma 3 your CPU is being choked and with it being choked with a 2nd pair of hands, your stream quality dies as well as game performance.
Personally its night and day experience for me going between X264 and NVENC on games like Arma, i use default settings 2600 on Main, 720P, Lancoz filtering all at 30fps on stream.
And she looks almost perfect, the only motion blur that can be spotted is when your running fast through foliage, other then that its perfect, frames are unhindered in the game, stream running stable and looking great, and CPU and GPU usage are both at optimal levels for the current load.
I hope this helped some new aspiring streaming with fire in his heart. Cheers.