Question / Help OBS 23.0.0 Encoder Overloaded

[c0nsecro]

After a lot of trouble with using OBS v23.0.1. and the new NCENC-encoder (resulting in ecoding overload in a lot of games with the same settings as in the old NVENC-codec) I've switched back to the old NVENC-encoder. The problems occured with only using the streamtest, without recording too. Also found out, that my i9-7900X and NVIDIA GTX1080 are not strong enough to handle 2715x1527 DSR, record it in the same resolution and stream it with 6000 kbits in 1280x720? The recordings looking choppy, even if the FPS are stable, the stream looks fine.

P.S. The last dropped frames are the result from the Fallout76 loading-sequences, maybe programming for multicores or I should raise the CQP level from 15 to 16.

 

[elbeegaming]

This is currently happening to me as well! I was wondering why no matter what my settings were, in game and in OBS, I was getting the "encoder overloaded" message every single time. I switched back to the regular NVENC encoder, and had ZERO issues. I assume it's some kind of bug? I have tried everything everyone has stated thus far:

- cap frames
- changing preset to "Quality" instead of "Max Quality"
- lowering bitrate
- unchecking "enforce streaming service encoder settings, look-ahead, and psycho visual tuning"
- turning v-sync on

I have no idea what's causing these issues, but I am not using the new encoder until this gets figured out.

 

[Fam3mon5ster]

hmm must be a bug then - I've been running into the same issue since beta although my new nvenc recordings / streams seem to have resolved now as of yesterday after installing the division 2 open beta? - im not too sure what happened but out of the blue i wanted to go trigger my emotions again since i just cant seem to register why the new nvenc doesn't simply work for me but with the old one i have 0 problems. did a recording on old and new nvenc on division 2 and im once again baffled.. after review of both logs, obs reports no issues with my recording which sparked my interest. checked play back and both recordings are perfect- I went back to run my benchmarks through gears of war 4 which is where I was able to replicate the drop / skipped / lagged frames in and surprisingly i got no issues old & new -

My guess is when I remember seeing division 2 installing , i saw it reinstalling direct x - my guess is this resolved it or could be a window that i may have not been running like i usually do ? i do remember launching obs studio under old nvenc and have the display window turned off ( similar to the slobs performance mode ) - changed to new nvenc and simply just hit record - those are the last things i remember i did last night. some how got it to work correctly - when i get home later ima see if its still resolved or back to doing its bs

 

[Tiyo]

I also notice you have replay buffer AND streaming going at the same time? Is this also happening in SLOBS? Because if not, then that could be why you have different results between the two. Try disabling the replay buffer all together and go again.

Isn't replay buffer RAM dependant rather than GPU/CPU?

 

[MadVaper]

I posted about this several days ago. I have the same issue. What worked for me was turning off anything that uses CUDA. So uncheck Pycho Visual and Look-ahead, set B-frames to 0, use the Quality preset or lower, and make sure your downscale filter (under Video) is Bilinear. This looks marginally better and uses fewer resources compared to old nVENC, but that's subjective. Your results may vary.

Still would like a proper explanation and/or fix for this. Even when GPU and CPU have plenty of headroom, I get massive rendering lag and encoding overload.

 

[Ch0ke_tv]

Render lag is because your GPU is maxed out. Why it happens in OBS but not in SLOBS I don't understand, but that's the cause. OBS needs at least some of your GPU % to be able to draw the picture.

I also notice you have replay buffer AND streaming going at the same time? Is this also happening in SLOBS? Because if not, then that could be why you have different results between the two. Try disabling the replay buffer all together and go again.

not sure if you were replying to my original post or someone else in the thread but I do not have replay buffer enabled and it still gives the encoder overloaded message.

 

[Agamemnus]

Your logs say you are. Try disconnect the hotkey from recording and replay buffer? Also, you might wanna check D:/R3corDinG for a bunch of files taking up disk space!

 

[Fam3mon5ster]

hmm must be a bug then - I've been running into the same issue since beta although my new nvenc recordings / streams seem to have resolved now as of yesterday after installing the division 2 open beta? - im not too sure what happened but out of the blue i wanted to go trigger my emotions again since i just cant seem to register why the new nvenc doesn't simply work for me but with the old one i have 0 problems. did a recording on old and new nvenc on division 2 and im once again baffled.. after review of both logs, obs reports no issues with my recording which sparked my interest. checked play back and both recordings are perfect- I went back to run my benchmarks through gears of war 4 which is where I was able to replicate the drop / skipped / lagged frames in and surprisingly i got no issues old & new -

My guess is when I remember seeing division 2 installing , i saw it reinstalling direct x - my guess is this resolved it or could be a window that i may have not been running like i usually do ? i do remember launching obs studio under old nvenc and have the display window turned off ( similar to the slobs performance mode ) - changed to new nvenc and simply just hit record - those are the last things i remember i did last night. some how got it to work correctly - when i get home later ima see if its still resolved or back to doing its bs

so did some more testing found out that the New nvenc is working for me now - tried to find the worst scenario to get the nvenc to lag -- gpu is capped apparently at 98% usage, cpu sits in the 40% when running gears of war 4 benchmark ultra everything-- spotify running playing music - also twitch app was on watching a live channel - also want to mention im running display capture only ( worst scenario) vs game capture -- ran 1st test no issues - everything fine then started introducing other factors -- found out --- when i had firefox open though running the same live channel i was watching in the twitch app i lagged 1.3% & skipped frames 21.7% (timestamp 20:53:42.437) closing - fixed this by BY TURNING OFF HARDWARE ACCELERATION - reran same channel no issues (timestamp 20:59:01.585) - also the obs preview screen while active caused it to lag 0.7% and skip frames 0.7% ( time stamp 21:00:51.634) turned it off no lag or skip (21:02:50.282). introduced webcam - no issue there either messed around with other active scenes that are not static - still no issues - You can see these results under the log ending in 25-

for anyone that has run gears of war benchmark knows there is a "insane" setting - under depth of field and screen space reflections - with everything running as previously mentioned working in the background that say every streamer may have going - i decided to turn the 2 insane settings on - ended up pushing the gpu to 99% and caused the nvenc to lag 7.7% and skipped 56.8% (timestamp 21:22:36.903) ---- gets worser when i singled out specifically "screen space reflections" lagged 14.1% skipped 67.4% (timestamp 21:26:33.508) -- going back to back to ultra on screen space reflections and leaving depth of field at insane showed no lagged or skipped frames -- found in log ending in 00

to sum ish up - dont have preview screen "live" on obs - if running a browser like firefox - try disabling hardware acceleration to offload gpu usage and put it on your cpu ( ** if u have cpu usage to spare ) for some reason firefox seems to hog alot of resources. - most modern games have "screen space reflections" as a option now - u may wanna mess with it see if u can alleviate some gpu resoruces in my case - ALOT lol - for anyone that cares- amd 2700x - 2080 ti - 3200mhz corsair vengeance pro ddr4 cl14 -- pushing a 3440x1440 ultrawide screen - gsync and vsync on - frame limited to 97 as its a 100hz monitor -

 

[c0nsecro]

View attachment 43706

Your logs say you are. Try disconnect the hotkey from recording and replay buffer? Also, you might wanna check D:/R3corDinG for a bunch of files taking up disk space!

Well, that's my log, I'm using the replaybuffer. But haven't problems at all before.

 

[SkiaTheShade]

This is happening to me as well with OBS 23.0.1 and the new NVENC encoder on an RTX 2070. I also play at 1440p/144hz with G-Sync. I realized that I had Plays.tv running in the background, which uses NVENC as well, and closing that down helped out A LOT. That being said, the problem still exists. Glad I'm not the only one!

 

[Ch0ke_tv]

The only way I can stream at 60fps (720p and 1080p) is to limit my fps in game to 60 or play at 1080p (I have 1440p native monitor), which is fine in some games, but horrible in others. I have disabled look ahead and psycho tuning, I even lowered the quality preset to performance and it still overloads encoding even at 720p60fps... Just for sh*ts and giggles, I tried x264 medium and was able to maintain 1080p60fps without overloading, however my in-game performance was significantly impacted in every game I tried, some to the point where it was not playable... How is it that OBS 23.0.1 is supposedly optimized for NVENC yet SLOBS seems to use far far less resources? Something doesn't make sense.

 

[Ch0ke_tv]

So I tried using the old NVENC encoder at 720p60 and 1080p60 and I still get Encoding overloaded errors unless I cap my framerate in game... I do not understand why SLOBS can use NVENC at 1080p60fps with no in-game frame caps and not have any issues but this new "optmized" OBS can't, even when using the older encoder... It doesn't make sense. There is 100% a bug with OBS 23.x.x. and NVENC encoding.

 

[MASTERkEN316]

My System:
i7 8700k @ 4.9ghz
32gb DDR4-3000mhz RAM
MSI DUKE RTX 2080 (419.17 driver)
1Tb SSD.


I am trying to stream at 1080p60fps using the new OBS 23.0.0 for the enhanced NVENC encoding quality, however I am constantly getting encoder overloaded and dropping frames hard. I have attached a picture of my output settings. I have also attempted to stream with Look-ahead and Psycho visual tuning disabled and it made no difference. Also, changing the quality preset to quality or even performance still has no impact. I have even tried capping my FPS in game to give the GPU more headroom.. still getting encoder overloaded message. I have no idea what the issue is but I am able to achieve 1080p60fps streams with no dropped frames in SLOBS just fine, but not in OBS Studio... So much for OBS v23.0.0 being optimized for RTX encoders.. or am I missing something? I will be going back to SLOBS until I figure out how to fix this.

Thanks for your help.

If u upgraded the software u will have to run the connection , auto wizard. Check all of your video settings. You may be recording at too high a video resolution, for the frame rate. Run auto connection wizard.

 

[Ch0ke_tv]

If u upgraded the software u will have to run the connection , auto wizard. Check all of your video settings. You may be recording at too high a video resolution, for the frame rate. Run auto connection wizard.

I am not recording. Just streaming. If you read through this thread, you would know that I have tried 720p and 1080p, both overload encoding. SLOBS works just fine at both resolutions. It's only this new "optimized" NVENC OBS that gives me problems. I have re-installed the software multiple times and ran the connection wizard each time with no difference.

 

[Clavius]

Do you try to turn off the preview in OBS? If I turn it on, encoder sometimes overloaded when using the new NVENC.

 

[Ch0ke_tv]

Do you try to turn off the preview in OBS? If I turn it on, encoder sometimes overloaded when using the new NVENC.

Yes I have tried that. Still overloads... The ONLY way I can get it to not overload is to either cap my FPS in every game to 60, or stream at 30fps... even lowering the quality preset in OBS to performance I still get encoding overloaded errors which pretty much confirms that the software is bugged.. I even lowered the graphics settings in game to Medium, still overloads. There is absolutely NO reason why it should be overloading on such a minimal quality preset. I attached a screenshot that includes the OBS window, game running and task manager showing the GPU encoder at only 5% usage but being overloaded... Something doesn't make sense here especially since SLOBS works just fine streaming and recording simultaneously at 1080p60fps... Something is bugged in OBS 100%.

 

[Ch0ke_tv]

Max performance preset, no look ahead or psycho visual tuning, 720p60 resolution (output), 1440p High (AA disabled) in game and still overloading encoder... OBS is 100% broken. I also tested with Apex Legends and Metro Exodus with the same results.

 

[ArhangelSkr]

I join, the same problem. On GTX, there is no problem, the problem is on RTX. The answer from the representatives will be?

 

[c0nsecro]

Did the problem still occur with the new update 20.0.2 too? Haven't the time to test it again until now ...

 

[ArhangelSkr]

Hike developers scored a bolt on this problem.