Question / Help AMD VCE Encoder sucks. Is Nvidia reliable for the task?


New Member
Good day, everybody.
I've been dealing with this issue for far to long and I'm growing tired of this.
I wanted to make this post simple, but half way through it, I figured it's not that easy. Here's my specs:
  • OS: Windows 10 Pro x64;
  • CPU: Intel i5 7600K 3.80GHz;
  • GPU: ASUS Radeon Rx 580 8GB;
  • Motherboard: Gigabyte H270-Gaming 3;
  • RAM: Corsair Vengeance 16GB (2x8GB) 3200MHz;
  • Storage: Corsair MP500 M.2 240GB + WD 3TB 7200RPM.
Not to go into too much detail, recording with this beast is a task that always worries me. I am able to record Overwatch at 1080p 35k bitrate flawlessly, but games like Dead by Daylight, Castle Crashers, Hunt: Showdown and even Axiom Verge would I encounter the infamous Encoder overloaded message. Needless to say, it ruins my footage the moment it pops up, it's unusable and I can't believe I'm encountering such an issue.
I have a sort-of workaround, and that's by using my iGPU's QuickSync encoder, which has worked fine, but I always fear that it would overload at some point. You can say that the iGPU works independently from the CPU, but I feel like that it will drop on me at any moment, and I want to go safe than sorry. Also, I have to uninstall it in order to be able to edit the footage later, which is a huge pain in the ass.
I've been doing research for over a month already, I've tried many things to get this working, bringing a list from a previous post:
  • Disable antivirus while recording. Didn't work;
  • Disable Windows Defender. Already done that, didn't work;
  • Disable Firewall. What does it have to do with anything? No, it didn't work;
  • Lower the recording quality. As much as I hate doing that, I did, and it didn't work either;
  • Record using x264 encoder. No, I already give the CPU enough stress with the highly-demanding games. Also, I can't reach the same level of quality without over-killing it, that's why I decided to go with H264 (and if that's not true, then prove me wrong);
  • Record using H265 encoder. Didn't work;
  • Don't record those games. Yes, I've been told that;
  • Use Game Capture instead of Display Capture and don't have them both on the same scene. Done that, didn't work;
  • Reinstall the OS. Done that already, as tedious as it sounds, but didn't work;
  • Change OBS's Process Priority to a higher one. Didn't work;
  • Change the game's priority to below normal. My computer isn't a paperweight, it can run Final Fantasy XV at High just fine. That being said, it doesn't always work;
  • Upgrade my PC. Seriously? Have you seen the beast I have under my desk? (no sexual euphemism intended);
  • Use another recording tool. I wish, but doesn't seem to be the solution either. I tried Bandicam with H264 encoding, works wonders (except for Dead by Daylight), I tried ReLive and it works perfectly with every game I play, but they don't cover my necessities. I would move if there was one that lets me record a video at any resolution I want (I record at 1920x1800), lets me record with more than 2 audio tracks without the need of any complicated setup and record my webcam separately (hence the resolution), all at once. No, XSplit is not an option;
So... as you can see here, I have my reasons to believe that AMD's VCE encoder is awfully unreliable. Not to mention, I have another issue of my own with AMD's latest drivers and it hasn't been solved yet. I have faith it will be, but by the time they do, it'll be too late, I am already very upset with the delivery. And I'm asking you guys:
Is Nvidia's NVENC encoding more reliable? Can I trust it?
I'm being serious here, I have my reason to not trust AMD anymore and I want this to change for good. I have found a GPU that is almost at the same level as the one I currently have, a GTX 1060 6GB, which I didn't buy because the one I have was on sale back in the day.
Also, I've seen most (if not all) of the biggest YouTubers out there using Nvidia GPUs only. With the demand they have, I'm pretty sure they went through the same issues I am at the moment. I'm also sure that they don't have these problems anymore and I want that, no more issues.
I'm not asking to be able to record footage at 4K 144 FPS, I'm simply asking for 1080p, 60 FPS and the best quality I can achieve without caring for the size. Having QuickSync might be fine and dandy, but as I said earlier, if I don't uninstall it before I start working on the videos, the rendering goes bad and I haven't found any solution other than that.
Can Nvidia do it? I am about to buy the GPU I mentioned before, but I don't want to throw my money away on something without your opinion.

Thank you very much in advance!