When running a game at super high settings, your card doesn't have a lot of power left to put into the encoding
That alone explain why
it's worth using a second card. It's been a while since I wrote in this thread...
I tried QuickSync for some time and for whatever reason, it didn't work for me, it should, I think... I mean, I know it should, but it didn't, so I gave up.
Using NVENC from my GTX 1080 while gaming (YES, don't be a prude, a GTX 1080 is not like some overpowered God), being that game taxing on the card to render all those gorgeous 18X heavy-dutty-AA on textures and shadows, could impact performance. It actually did, it happened. Some games can use 100% of your card, so what's left for encoding? Think about that for a moment... Something gotta give...
So I had a GTX 750ti to test and, oh boy, it lifted the heavy encoding task from the GTX 1080 like a boss. And old card, probably inexpensive in some countries...
But I couldn't dedicate the GTX 750Ti for that, so I bought a a RX 460 4GB OC from Gigabyte, probably another inexpensive card in some countries. And oh boy, I don't know why but AMF + OBS is actually great. Like if you know what you're doing, even the low end model, the RX 460 has A LOT of options to fine tune inside OBS.
Now, keep in mind, I was using that for capturing the game at this point, not Twitch or some other live service. So it was good, I captured at absurd 50.000 bitrate, fooled around with QPS at 22 and it was great. But then I decided to stream.
So let me tell you my 50 cent about streaming with a encoding API on you graphics card: it's not great. I'm not technical, but putting it simply, not using technical jargon, it's like the quality per bitrate is better when you use the CPU, even when you use that veryfast profile. And now that I started streaming, I went live, I noticed that QuickSync can actually do something, it can encode for stream, since I was streaming at miserable 4.200 bit rate.
So I got a Ryzen 2700X now and I use that for streaming, 720p@60, 4.200 bitrate and the AMD card is used for capture.
Other thing I'm doing is keep OBS open on a display connected to the RX 460 (before the Ryzen 2700X upgrade I used the intel GPU for that, so the RX was also not bothered with that), so not even the OBS Preview is rendered on my GTX 1080 and yes, that's actually a thing, I monitored the cards and you can actually see how OBS Preview use a good percentage your card.
UPDATE, forgot to mention something.
Another thing that I tried was OBS to OBS from one computer to another, AMF with the RX 460 on the gaming PC to a second PC using x264 on a i7 4770K over wired gigabit LAN. Technically it should work. The results were not good. I tried that using that nginx+RTMP trick someone wrote on the forum. Something I think Linus from LinusTechTips is doing with their live video, but they probably took it to a whole new level.
Again, I'm not technical, I'm practical :)
So this is just an update to the thread, resuming my experience living through this debacle myself :)