The quality of the x264 "veryfast" preset is about the same as the quality of nvenc on the GTX 10x0 cards. This is some kind of break even point. If you have a CPU that is able to handle a better preset than veryfast, such as faster, fast or medium, you create better quality with x264. This is mostly true for streaming PCs where there is no game to run at the same time. Most better desktop CPUs are able to achieve this.
With the new RTX 20x0 cards, the nvenc encoder was improved. It is said its quality is between fast and medium x264 preset. The break even point has shifted with that, so it is often better to just use a single desktop PC for gaming and streaming at the same time and use nvenc instead of x264. OBS explicitly supports a single PC setup with the nvenc (new) encoder, which avoids copying raw video data over the pci-express buss. This is true for every nvenc system, not only RTX setups.
Now let's talk about the GPU in the streaming PC. OBS still needs a minimum amount of GPU power to composite the video. Only with trivial scene setups and small canvas, an iGPU is able to handle this. For a more sophisticated setup with multiple sources, multiple scenes and the usage of filters, most iGPUs are overloaded. For such a setup, you need a "real" GPU even on a streaming PC. Your additional 1080 Ti is somewhat overkill, but it is not completely wasted in a streaming PC because of this.
The best price/performance ratio for a pure streaming PC was the GTX 1050 because of the nvenc encoder in case the CPU wasn't able to handle more than x264 veryfast. However, with the phasing in of the new RTX 20x0 and GTX 16xx cards with Turing chip, one might look for a newer version of a low-cost, yet somewhat powerful GPU. For some reason, I don' recommend AMD GPUs, but I might be biased. It's mostly because of the nvenc encoder, which is unmatched by the AMD VCE hardware encoder.
Based on my understanding of your reply, it sounds like you recommend I just stick to a single PC setup and upgrade my GPU to an RTX based GPU. Am I understanding this correctly?
Based on your comment on using CPU x264, would the new AMD 3900x be powerful enough to encode since it seems the GTX1080ti is equivalent to x264 very fast on CPU. Im seeing reports of people getting better results on gaming with the AMD 3900x by locking the game to use a single CCD chiplet (6cores/6threads) vs the entire chip, which would leave an entire another half of the CPU to be available to other tasks.... maybe encoding??? Example given on forums: 3900x using all cores with PBO, Battlefield V average FPS of 153, but when they locked the game to only using the 6 cores on the 2nd CCD on the CPU, they saw average FPS increase to 171 due to scheduling issues in windows trying to cross the infinity fabric instead of prioritizing use of a single CCD before pulling from the other chip...
Also while doing some reading on the use of NDI in OBS, I read its best to send the monitor's native resolution as the NDI resolution source to the 2nd PC, then downscale encode it to 1080p60 for least loss of FPS drop in gaming and least amount of CPU usage on the gaming rig. So in my case, I want to use my 1440p@165hz monitor for a gaming rig, so I would send 1440p source to 2nd pc, then convert to 1080p to send to stream aka twitch?
The 1660 is probably the sweet spot for anyone who doesn't require they pay bottom dollar for a card-- it's got the encoder chip from the 20xx series without the cost or the ray tracing.
So if I am understanding this correctly, it sounds like the RTX GPU is an upgrade over the 1080ti I have currently in regards to encoding on the stream pc (2nd)?
Lastly, I want to ask these questions..... If I was to pursue the 2 PC streaming setup.....
1) what is like the minimal CPU that could actually encode 1080p60 on its own without a GPU, ask this because I'm seeing a lot of people dumping 1st and 2nd gen Ryzen hardware right now. Last night I saw an AMD 1700 for sale on eBay go for $81 shipped.. and an auction on an AMD 1900x threadripper CPU end at $117 shipped. Unsure how much CPU power would be needed for CPU only encoding....but maybe a single CPU/Mobo setup would be the cheaper route....???? The benefit of this route if better would be i could use m-itx mobo in my system and since they only come with 1 single PCIE slot, I could use a 10GB NIC, since I've read its highly recommended for NDI encoding.
2) Doing a little reading and research, I came across EposVox who went the route of Quadro Card for GPU encoding. He mentioned in his setup that the minimum hardware needed for his goal according to Nvidia matrix sheet was the 1050ti, but couldn't find one for a reasonable price at the time of his testing/purchasing, but he found a brand new Quadro P400 for around $100. In his testing, he stated he was able to encode with it 1080p@60 "high quality".... so not really sure where that stacks up on the chart for stream encoding compared to let's say the 1080ti, since he mentioned the Quadro is more efficient at doing encoding vs a typical GPU..... I ask this because there are a handful of P400 cards used selling in the $35-50 range on ebay.