Question / Help NVENC H.264 Encoding Overload Issue

[Devin_94]

Can anyone help me with this issue I'm having while recording or streaming for some reason obs is giving me an encoding overload error with randomly started happening a few days back. I'm sure my pc hardware is capable of handling the setting I have (keep in mind i am not gaming on my pc just using it to record my xbox gameplay).

PC Specs (no overclocks on any items):

i7 8700K
EVGA 1080ti
16gb Ram
Elgato 4k60 Pro
850W PSU

I have tried everything i can think of to solve this issue but nothing seems to work any help would be greatly appreciated (log file will be attached below).

 

[koala]

You stream with 1920x1080 and record with 3840x2160 at the same time, thus resizing the video from 3840x2160 to 1920x1080 in the encoder settings. This is a somewhat cpu-intensive and slow resizing, so this might be one cause for a performance issue.
Then you're using nvenc twice in parallel, one session is encoding 4k and the other is encoding hd, both 60 Hz. It might be that this is too much for the nvenc encoder.
Try to use x264 for streaming and keep nvenc for recording.
If this doesn't work, try to find out about how the system behaves if you do only one encoding session. if you stream only, change Settings->Video->Output to 1920x1080 and don't resize in the output settings. This will make OBS use the GPU for resizing.

Final comment: con't use CBR for recording. You're wasting bandwidth and sacrificing quality. Use CQP with a CQP value of 18-23 (lower value means higher quality). And don't use mp4 as output format but *.flv or *.mkv instead. An mp4 is destroyed if you don't gracefully terminate recording.

 

[Zidakuh]

First up, don't record with CBR. Secondly, if recording to a HDD and not an SSD, the drive might not be able to keep up with a high bitrate.
Try using CQP 15-21 for recordings. (lower value = higher quality)

 

[carlmmii]

Turning off 2-pass encoding should take a huge load off the recording encode. The stream encode doesn't appear to be running into any issues (doesn't the resize operation happen on the GPU if it's done for an NVENC encode session?)

Also... it doesn't look like it's interfering as of now, but Game DVR is on. This adds its own NVENC encoding session when active, which can prevent one of your OBS encoding sessions from starting (limit of 2 simultaneous encoding streams for consumer cards).

 

[Devin_94]

I appreciate the prompt responses but even if I just record a video in 4k I have the same issue of encoding overload. Before I had this issue I could record and stream at the same without issue. But now no matter what I do I cant get this issue to correct. I will try and do the above mentioned recommendations and will keep this thread updated.

 

[Devin_94]

Alright here is and update i changed some setting around but found that i was still having the same issue granted it did not happen as fast but it was still there. i know it was recommended to me that i dont downscale my streaming resolution but the only issue i have with that is i dont focus on streaming i focus more on recording 4k gameplay (only stream when i feel like it with or for friends). is there any other changes i should make do keep in mine my goal is to stream and record in 4k at the same time may not be all the time but when i feel like it thats what im shooting for.

 

[carlmmii]

I'm wondering if you're having a bandwidth limitation just due to frame size. You might want to experiment with the beta version that has NVENC optimizations which allow it to send the rendered scene straight to the encoder.

 

[Devin_94]

Could it be that my hardware just isnt strong enough?