Question / Help Encoder overloaded while streaming and game primary window.

[Yodonashi]

Hello All,

I've run into an odd issue that I've been smacking my head against for awhile now and can't seem to figure out.

I built this PC for streaming in January and have had no problems streaming at 1080p60fps with Lanconz scaling and a 6000 bitrate until the past few weeks since version 26.0 and 26.1 were released. When I say no problems I mean 0 dropped frames, no encoding overloads, nothing. My CPU usage rarely ever peaked over 30% even.

Suddenly now though, it doesn't matter how low I set my settings, I ALWAYS without fail, overload encoding and drop below 15fps. What I've noticed however is that this only happens while whatever game I'm playing is the primary interaction window on my PC. If I alt tab to OBS or any other window, I immediately jump back up to 50fps, and drop below 10% CPU usage, even though OBS is still rendering the animations. The moment however that I interact with the game scene, I drop down to 15 or less fps.

I've done everything I can think of, even dropping down to 720p30fps with my scaled resolution and bilinear scaling.

It is worth noting that this happens only while streaming, it does not happen while recording. I've bounce between both x264, and NVEC encoding hoping it would make a difference, but it has not.

I've attached my most recent log file for any help anyone may be able to give me. It would be greatly appreciated!

 

[Rdelaura]

Same for me. Tried everything.

 

[Rdelaura]

Ever since V23.0 was released which was around the same time as a Windows update was released and a GPU driver I've been having this problem. Was perfect and flawless prior to this. No matter what game I play I cannot hold 60 fps even though in game im hitting well above 100+ FPS. When I click out of the game the fps in OBS holds to 60 but when I click back in game it dramatically fluctuates between 40-60fps

My hardware is pretty darn good and should not be having any problems

I7-6850k
Strix 1080TI
Strix X99 mobo
32GB Corsair Platinum RAM
500 Mbps downstream
50 Mbps Upstream

Game running at 2560x1440 @ 100+ fps
Stream downscaled to 1600x900 @60fps

Tried both Nvenc encorders with the old one having better performance (new has GPU overload problems)
CBR @ 5500
Tried Max Quality -> Performance
Two-pass disabled
Psycho Tuning disabled (tried both)

I've tried rolling back OBS, Nvidia drivers, Windows update..... nothing helps.
I either have to find a different program to stream from or just not stream anymore because this makes no sense and there is no help. I've posted my logs before but because nobody seems to respond on here whats the point?

 

[carlmmii]

This all stems from windows prioritizing the game over OBS, causing rendering lag. There's several factors that you're running into:

Game mode is on. You need to update to windows update 1809 or later, and then disable game mode using these instructions.

(possibly) game framerate is uncapped. If you're letting your game use all available resources even past what your monitor can display, then you're likely starving OBS of the resources it needs to perform scene rendering. Cap your framerate in-game (either by setting a max framerate, or enabling v-sync).

For the encoding lag the logs are reporting, that's a side effect of the new nvenc encoder and specific settings you're using. Change "Max Quality" down to "Quality", and turn off Psycho Visual Tuning. Both of these options use extra GPU resources to help improve encoding quality, but can hinder performance if OBS doesn't have the available resources. This is specific to the (new) nvenc encoder, which was just introduced into version 23.x.x in march... which could possibly explain the recent issues.

 

[Rdelaura]

Thanks for the response and I know a log would help which I would upload but am at work currently

Game Mode is NOT ON
I've tried to limit fps in game to 165 but makes no difference. I'm not dropping to 60 fps in game when my monitor is 165hz, whats the point of the high refresh monitor if I cant use it fully
I've started at Max Quality and went down to Performance with slight improvements but still problems w/ Psycho Tuning Off

If my hardware or internet was crap I'd say it was that but there is no reason for this.

Old encoder works better but still cant hold 60 fps

Like I said I was flawless prior to 23.0 update (rolling back didnt resolve anything)

 

[carlmmii]

My response was directed toward the OP. Please post a log file of your own, everyone has a different specific set of hardware/settings/issues.

Also, "high refresh monitor" and "smooth capture" are unfortunately usually mutually exclusive. Windows doesn't like mixing framerates, especially between monitors. Also, capturing/encoding is a very demanding process which can't just be assumed to work because you want all the frames.

 

[Rdelaura]

Well like I said... flawless and zero problems for a long time prior to the March OBS/Windows/Nvidia update
Flawless

 

[Rdelaura]

There the last log I have available at work if you want to take a look at it.

 

[Narcogen]

16:12:53.019: Windows Version: 10.0 Build 17134 (revision: 523; 64-bit)

Windows is out of date and Game Mode is on by default and cannot be turned off. (You used to be able to turn it off by registry editing, but if you had done so, OBS would note this in the log, and I don't see that here:

16:12:53.020: Windows 10 Gaming Features:
16:12:53.020: Game Bar: Off
16:12:53.020: Game DVR: Off
16:12:53.020: Game DVR Background Recording: Off

At the bottom there it would say "Game Mode: Off" but it doesn't.

The link below has more information.

 

[Rdelaura]

16:12:53.019: Windows Version: 10.0 Build 17134 (revision: 523; 64-bit)

Windows is out of date and Game Mode is on by default and cannot be turned off. (You used to be able to turn it off by registry editing, but if you had done so, OBS would note this in the log, and I don't see that here:

16:12:53.020: Windows 10 Gaming Features:
16:12:53.020: Game Bar: Off
16:12:53.020: Game DVR: Off
16:12:53.020: Game DVR Background Recording: Off

At the bottom there it would say "Game Mode: Off" but it doesn't.

The link below has more information.

This is in reply to me or the OP?

 

[Rdelaura]

If you were replying to me and if this works.... I seriously will love you forever... and ever

 

[Rdelaura]

Alright you were talking to me. I could have swore I had that turned off but yea I dont think in the registry so thank you I will take a look at it after work. I cant believe nobody from previous logs has mentioned it. OMG, you make me wanna take a half day and go home and see if its this.

Also in terms of Windows being updated. Around that same time I DID update Windows a while ago but it failed and everytime I start up Windows it says its installing updates, then fails and reverts, restarts and is fine. Just never updates. I've tried everything recommended including repair but nothing seems to fix it. I was told to use my "recovery disk" and repair Windows but I installed off a file and flash drive so I have no idea how to do that. Like I said, Im hoping its the registry Game Mode thing because you would make me so happy

 

[WizardCM]

Hi guys, I've had a chance to skim through the issues you're having, and some of the things Narcogen mentioned are no longer relevant. I'll provide as much detail as I can below:

• Windows 10 does prioritize the currently focused game/application over others, especially when it comes to GPU. OBS itself tries to raise its priority on the CPU, but Windows has no ability for OBS to request more GPU. This should behave a little better in Windows 10 1809 and even better in Windows 10 1903.
• Game mode when ENABLED in Windows 10 before 1809 definitely lowered the performance of OBS. The core purpose of Game Mode is to more effectively prioritize the performance of your currently running game, including lowering the priority of other applications (like OBS) and even closing extra background services. Overall, this is a good thing, but was causing side effects with OBS. However, as of this update, this is no longer the case and we recommend leaving Game Mode ENABLED going forward (1809 and above).
• The other week, NVIDIA released a GPU driver that caused significantly degraded performance across the board. They quickly released a hotfix, however your computer may not have downloaded/installed it automatically - please make sure to update.
• In Windows 10, if you have more than one display, and the primary display is at a refresh rate higher than 60Hz, putting OBS on a secondary 60Hz display will degrade performance in both the game and OBS, regardless of whether OBS is capturing it. This is caused by DWM and hardware acceleration. The same would happen if you had OBS closed and instead played a YouTube video in Chrome or Firefox on the secondary display. My understanding is that Microsoft is looking into it, but nothing to share at the moment.
• If you're using the (new) NVENC encoder, which keeps everything on the GPU instead of copying to system RAM, make sure your GPU usage stays below 90% - it is a lot more sensitive to high usage on that front. If you're using Advanced Output mode, you can try switching to the old NVENC encoder to see if it helps.

 

[Yodonashi]

Hi guys, I've had a chance to skim through the issues you're having, and some of the things Narcogen mentioned are no longer relevant. I'll provide as much detail as I can below:

• Windows 10 does prioritize the currently focused game/application over others, especially when it comes to GPU. OBS itself tries to raise its priority on the CPU, but Windows has no ability for OBS to request more GPU. This should behave a little better in Windows 10 1809 and even better in Windows 10 1903.
• Game mode when ENABLED in Windows 10 before 1809 definitely lowered the performance of OBS. The core purpose of Game Mode is to more effectively prioritize the performance of your currently running game, including lowering the priority of other applications (like OBS) and even closing extra background services. Overall, this is a good thing, but was causing side effects with OBS. However, as of this update, this is no longer the case and we recommend leaving Game Mode ENABLED going forward (1809 and above).
• The other week, NVIDIA released a GPU driver that caused significantly degraded performance across the board. They quickly released a hotfix, however your computer may not have downloaded/installed it automatically - please make sure to update.
• In Windows 10, if you have more than one display, and the primary display is at a refresh rate higher than 60Hz, putting OBS on a secondary 60Hz display will degrade performance in both the game and OBS, regardless of whether OBS is capturing it. This is caused by DWM and hardware acceleration. The same would happen if you had OBS closed and instead played a YouTube video in Chrome or Firefox on the secondary display. My understanding is that Microsoft is looking into it, but nothing to share at the moment.
• If you're using the (new) NVENC encoder, which keeps everything on the GPU instead of copying to system RAM, make sure your GPU usage stays below 90% - it is a lot more sensitive to high usage on that front. If you're using Advanced Output mode, you can try switching to the old NVENC encoder to see if it helps.

Thank you very much for the extremely informative reply. I am currently installing the hotfix as we speak.

As for the secondary display issue, is this still a problem for users if I have hardware acceleration disabled? If so how might I go about capping my refresh rate at 60 on my displays so as to mitigate this problem?

Lastly I've tried both encoders to no avail. There really isn't a concern for hardware with the capabilities they have, 9th gen i5, 1050ti, and 24GB DDR4 so there shouldn't be a hardware issue.

 

[Rdelaura]

Alright man, so I got a constant 60fps (thanks, I think it was Game Mode) although encoder overload. I only tried the NEW Nvenc & the old one was working for me so tomorrow Im gonna test the old encoder. Thanks again, its way better than it was

 

[Rdelaura]

@WizardCM Thank you so much for the helpful info. I rolled back my video driver because that hotfix didnt seem to help but im thinking it was something else so im going to redownload.

 

[Rdelaura]

I've updated to 1809 last night and switched back on Game Mode. Will be testing tonight and relay the results w/ log
Thank you to everyone for commenting and providing valuable information.

 

[Rdelaura]

So Im pretty sure the updating to 1809 helped a ton because I can hold a steady 60fps now although Im getting encoder overload even on the old encoder which causes stuttering and I also dropped my stream completely, then it reconnected. If any of you have the time to look at the log and if you notice anything I'd really appreciate it.

 

[carlmmii]

Are you sure you're updated to 1809? The logs seem to think otherwise, as it's reporting 17134 still.

19:01:37.164: Windows Version: 10.0 Build 17134 (revision: 523; 64-bit)

Besides that, I'm noticing a few things...

From what I can tell, as far as streaming goes, the main problem you're having right now is dropped frames. You can try lowering your bitrate to see if that helps. It's also showing some frames lost from rendering lag, but honestly that's expected with as many media sources that you're using (also, <0.1% is barely noticeable -- that's basically just a few skipped frames caused by media loading).

The encoding overload issue you have is specific to your recording. I would take a look at your GPU Encoder usage while doing stream+encode (just pull up Task Manager and go over to the Performance tab, select your GPU, and pay attention to the Video Encode plot). If you're reaching 100%, then that means you're overloading the encoder. I doubt you are though, considering the 10 series should have no problem running 2 instances of 900p60.

It appears you are still using the old Nvenc encoder though. I'm not sure if this is intentional, or if it's silently reverting back to it without your knowledge, but I would try using the new encoder to see if it makes any difference (that would eliminate a lot of PCIe bandwidth usage between render and encode, which could be the root cause here). I'd make sure to avoid Max Quality, Psycho Visual Tuning, and Look Ahead, just to eliminate any other possible GPU overhead.

 

[Rdelaura]

The new Nvenc encoder ALWAYS gives me overload.
I updated to 1809 last night, sat here until midnight when it was complete. Everything IS running better since. I will try to check again in the morning.