Question / Help Encoding Overload while streaming/recording on high end Streaming PC

[ProHenis]

I purchased a separate streaming PC 5 months ago to help with streaming and recording in high quality, and for a long time, everything was working perfectly; smooth streams and recordings. For the last month, I've been experiencing encoding overload when streaming & recording simultaneously in OBS Studio. When I first started noticing the problem, the encoding overload messages seemed intermittent. However, over the last week, I've noticed that I'll get my first encoding overload message about 20-30mins into the start of my stream, followed by more frequent encoding overload messages about 5 minutes apart from each other. This then continues usually for the rest of the stream. Each time I get the encoding overload message, both the stream and recording will freeze for 2-3 seconds and then return to a smooth 60fps. The encoding overload is bad enough that my viewers have come to notice it and complain.

The specs of my PC are good enough that I shouldn't be experiencing these problems. I've attached my most recent log file (https://obsproject.com/logs/Ftb9m1s0F4h42I6E) after a 2 hour streaming/recording session as well as the official specs of my Streaming PC.

Streaming PC Specs:

CPU - i7-8700K
GPU - NVIDIA GeForce GTX 1060 6GB
RAM - 16GB (8GBx2) DDR4-3000
Motherboard - ASUS TUF Z370-PLUS GAMING
OS - Windows 10 Home
Elgato Game Capture 4K60 Pro

Does anyone have any suggestions on how to fix this problem?

 

[Narcogen]

Okay, I want to make sure I'm reading this right.

You're recording and streaming simultaneously.

For recording, you're using the CPU x264 encoder and CBR rate control, and for streaming, you're using NVENC? Do I have that right?

 

[ProHenis]

Okay, I want to make sure I'm reading this right.

You're recording and streaming simultaneously.

For recording, you're using the CPU x264 encoder and CBR rate control, and for streaming, you're using NVENC? Do I have that right?

Yes, that's correct. I get the encoding overload message when I use the x264 encoder or the NVENC encoder for just streaming/recording, or both at the same time. For the first 4 months that I got the streaming PC, I was using the NVENC encoder for both streaming and recording and everything worked perfectly. I only started using the x264 encoder for recording to see if I could eliminate the problem but I still get the encoding overload messages. It seems as though the longer the stream lasts, the more frequent the encoding overload message appears and freezes the stream. Hopefully that all makes sense.

 

[Narcogen]

Okay.

You're sort of playing against the strengths of your hardware doing it this way.

NVENC encoders take load off your CPU, but generally deliver slightly less quality at a given bitrate than the software x264 encoder. The best use case for the NVENC encoder is local recording in high quality with large file sizes-- but-- CQP rate control rather than CBR, since CBR just inflates file sizes without adding any extra quality.

Streaming services require CBR and a much lower bitrate than local recordings, generally speaking, so the idea is to start with the veryfast preset and whatever Twitch Tester says you can hold as a stable upload bitrate and then adjust from there as needed; presets slower than veryfast increase load dramatically while giving only marginal quality improvements.

So when you were doing both streaming and recording on the NVENC encoder, if the quality was good enough for both the stream and the recording, that was probably absolutely fine.

If you're going to switch to using both methods, either to eliminate a problem or to try and improve quality, the best suggestion would be to swap them-- NVENC for recording, x264 for streaming.

Suggest making a new profile and starting over in simple mode with some of the presets and see how it goes, then adjusting from there.

 

[RytoEX]

To add to what @Narcogen said, you're streaming to YouTube, which will accept just about any incoming bitrate and then transcode the incoming feed. You can pretty much send them as much bitrate as needed to get a good quality picture. 7000 Kbps bitrate for 1080p60 is a bit on the low side. YouTube suggests 4,500 - 9,000 Kbps, so I'd push 9,000 Kbps or higher if your connection is stable enough to handle that. If you were streaming to Twitch, they're currently capped at 6,000 Kbps ingests, which is too low for high quality 1080p60 streams.

Most encoding/streaming guides give bitrate recommendations for x264, not hardware encoders. NVENC on the NVIDIA 10-series is pretty close to x264 in terms of quality/bitrate, so it should be viable on YouTube Live. I'd still add some extra bitrate if you have bandwidth to spare if you use NVENC to stream.