Question / Help Encoding Overload New NVENC with RTX

A

Aerith

New Member
Hello,

I have been streaming with SLOBS for a while but recently switched to OBS Studio after some friends convinced me. However, I keep getting encoder overloaded when I stream on the new NVENC for RTX cards and no matter what setting I change I keep having the issue. I am on version 23.0.0-rc2.
My specs are an i7 8700k at 5ghz, RTX 2070 EVGA Black, 16gb of ram. Any ideas?
 
SkiaTheShade

SkiaTheShade

New Member
I'm having the exact same issue here. No what matter NVENC settings I try I get that same Encover Overloaded message. My stream seems to start lagging pretty bad when it happens, too. I also have an EVGA RTX 2070 XC Black Edition, i7 9700k, and 16gb of 3200mhz RAM.
 
A

Aerith

New Member
SkiaTheShade said:
I'm having the exact same issue here. No what matter NVENC settings I try I get that same Encover Overloaded message. My stream seems to start lagging pretty bad when it happens, too. I also have an EVGA RTX 2070 XC Black Edition, i7 9700k, and 16gb of 3200mhz RAM.
Click to expand...
We have the same RTX card, is it possibly an RTX issue?
 
carlmmii

carlmmii

Active Member
Try turning off look ahead, psychovisual enhancements, and change from "Max Quality" to "Quality". All of these options use CUDA, which requires extra GPU resources other than just the NVENC encoder.

The other option is to lower the GPU usage of your game so that OBS has more to work with. Before, this would normally only affect rendering performance. Now, with the new NVENC encoding options, it can affect those features that leverage CUDA.
 
A

Aerith

New Member
carlmmii said:
Try turning off look ahead, psychovisual enhancements, and change from "Max Quality" to "Quality". All of these options use CUDA, which requires extra GPU resources other than just the NVENC encoder.

The other option is to lower the GPU usage of your game so that OBS has more to work with. Before, this would normally only affect rendering performance. Now, with the new NVENC encoding options, it can affect those features that leverage CUDA.
Click to expand...
Look ahead is off, I'm gonna try turning off psychovisual enhancements, I already have the quality set to maximum performance.
 
A

Aerith

New Member
carlmmii said:
Try turning off look ahead, psychovisual enhancements, and change from "Max Quality" to "Quality". All of these options use CUDA, which requires extra GPU resources other than just the NVENC encoder.

The other option is to lower the GPU usage of your game so that OBS has more to work with. Before, this would normally only affect rendering performance. Now, with the new NVENC encoding options, it can affect those features that leverage CUDA.
Click to expand...
Alright I gave your suggestion a whirl, still having problems. I tried capping my framerate in game too.
 
SkiaTheShade

SkiaTheShade

New Member
I also have Look Ahead and Psycho-Visual Enhancements turned off. I've tried multiple different quality settings and I have my in-game FPS limited to my monitors refresh as well. Interestingly, I never had this issue on my old GTX 970, 1440p screen, and with the previous NVENC encoder.
 
A

Aerith

New Member
SkiaTheShade said:
I also have Look Ahead and Psycho-Visual Enhancements turned off. I've tried multiple different quality settings and I have my in-game FPS limited to my monitors refresh as well. Interestingly, I never had this issue on my old GTX 970, 1440p screen, and with the previous NVENC encoder.
Click to expand...
Is it an RTX NVENC issue
 
N

NEE

New Member
Make sure you're running process priority as High in Advanced settings.

I tried almost all settings and was still getting the overload issue until I changed this. Now I'm running quality settings with no issues.

RTX 2080 Ti, i7 7700k, 16GB ram
 
A

Aerith

New Member
NEE said:
Make sure you're running process priority as High in Advanced settings.

I tried almost all settings and was still getting the overload issue until I changed this. Now I'm running quality settings with no issues.

RTX 2080 Ti, i7 7700k, 16GB ram
Click to expand...
Just tried it, no dice. It made things a bit better but video was still stuttering and getting encoding overload.
 
NordicBattalion

NordicBattalion

Member
You need to lower fps or in game quality enough so that the game doesn't use more than 90% of the gpu. OBS needs GPU
to render the stream. After that, the RTX takes over.
 
A

Aerith

New Member
NordicBattalion said:
You need to lower fps or in game quality enough so that the game doesn't use more than 90% of the gpu. OBS needs GPU
to render the stream. After that, the RTX takes over.
Click to expand...
I managed to get it to a watchable state, still stutters though. I can get the gpu down to about 80% usage but even then it's still not that smooth.
 
NordicBattalion

NordicBattalion

Member
Then you need to look at constant fps and stuff. If you don't mind the little extra input lag (i can't even measure it), lock the fps to like 100, but use vsync as well. In the top of OBS, go tools and choose upload log and link it here.
 
A

Aerith

New Member
Update... I updated to the official release of OBS 23 and disabled game mode in Windows 10 and I haven't had a problem since. I was able to stream for an hour with no problems whatsoever. If anything happens I'll be sure to post about it.
 
SkiaTheShade

SkiaTheShade

New Member
Aerith said:
Update... I updated to the official release of OBS 23 and disabled game mode in Windows 10 and I haven't had a problem since. I was able to stream for an hour with no problems whatsoever. If anything happens I'll be sure to post about it.
Click to expand...
I just updated OBS, and I'll be giving it a shot tonight! Hopefully the full release has fixed the issue, that would be amazing.
 
K

Koxicain

New Member
I have the same issue. 2700x RTX2060. Isnt there supposed to be a dedicated NVENC encoder on the card itself,
so you dont strain the GPU at all? So to lower the quality of games to give the GPU more headroom just doesnt make sense
unless you use Look ahead and Psycho enhancement.
 
D

Dirteebreaks

New Member
Its not just RTX, i have a GTX 1080 and have been using nvenc with max performance , psycho-visual and look ahead on 720p 60fps @ 5500 bitrate with no problems what so ever for months until yesterday!! I would get encoder overloading errors and the stream would go down to like 20fps or lower.

I have no idea what the issue is, so i used slobs and that would disconnect the stream but not say anything as if it was all fine, but i wasnt actually streaming lol.

I did just upgrade OBS to 23.1 But that dors not explain slobs crashing also.
Its sooooo annoying!
 
You must log in or register to reply here.
Top