Question / Help Encoder overloaded error when game is main focus but not unfocused.

[SailorDeath]

Encoding Overload on some games but not others. I noticed that if I load one of the games that's having this issue, regardless if it's full screen, windowed or borderless window if I have main focus on the game I get the error, if I select a different window the issue goes away. Also for some reason it doesn't do this with all of my games. For example, GTA 5 loads and I have no problems. Smooth graphics and no encoding problems. But if I switch to a game like Final Fantasy XIV, as soon as it loads I get encoder overload errors. Since I might have to alt-tab I have it set to maintain audio and full frames when I have alt-tabbed.

On saturday I streamed the game and this error was not present. Everything was running fine, however sunday night when I tried to stream the problems started. Nothing has changed on my system between those two days. However no matter what I try I cannot get it to stop lagging. In task manager I can see OBS going from it's normal 12% CPU useage and see it spike wildly between 0.8% up to 60% when I'm either streaming or recording video. I've tried changing the source resolution, the output resolution, putting the encoder on ultra fast, decreasing the frame rate, lowering the bitrate and nothing seems to want to work. But it never fails, if I change focus of the window to any other window but the game, it see it capturing at the target framerate with the CPU usage holding steady. I've also tried switching between software x264 and NVENC H.264 and I get the same problem. I'm hoping someone here knows what might be happening. Switching to DX9 helps a little but I still get CPU spikes that cause the message to display. It's so weird, I can sit in a high traffic area where a lot of effects are happening but if I don't have that window selected I can output a stead 60FPS at 1080P but as soon as I select the game window again it starts lagging.

You can see what the video is doing by going here, https://www.twitch.tv/videos/141024322 you can litterally tell when I select the game window again around 30 seconds in. I refuse to believe that it's an encoding settings issue with OBS because if that were the case it doesn't explain the lag when I select the game window.

 

[jakobmiller]

Same thing for me. Basically down to no settings what so ever and still overloaded. I could run at least 4500 bitrate, fast preset at 60fps/720p a week ago.

 

[Fenrir]

The only things I can think of that might be causing such drastic performance issues is any third-party "game boost" applications, or other process prioritization applications. Since it's only happening when the game is in focus, something outside OBS' control is preventing OBS from accessing the resources it needs to function properly.

 

[Calys_PewPew]

Hello,

I've had EXACTLY the same issue. Everything was working absolutely fine until today. I changed nothing, did no updates neither on Windows, on my CGU drivers nor on OBS.

Today I tried to stream different games and the problem appeared every time (I tried Rocket League, Little Nightmares, and Mario Kart 8 using my Elgato Capture HD). Whenever i'm on the OBS window, the stream is OK, but as soon as I switch to the game, I get an "encoding overload" error and the stream becomes laggy as hell. (My voice seems to be ok tho).

I desperately search for the solution for hours now ... :'(

 

[SailorDeath]

Can any of you do me a favor and try something, pull up a youtube page or some other page with streaming video (except netflix) This is more prounced on twitch streams for me but whenever I launch FFXIV while watching twitch it KILLS the video stream, I can still hear audio but the video itself is frozen, after a while it'll stop altogether and won't play. I'll just have that little buffering icon in the middle of the screen, Turning off hardware acceleration in the browser helps some but it's still pretty stuttery. I have a high end system and I've seen some people on potatoes not have this issue. It's almost as if the PC is putting everything program on the same core instead of having different cores handle the workload.

 

[SailorDeath]

okay, so I rolled back my driver from most current that I had installed on friday last week to the one I was using before that (version 381.65) and I can now use software encoding again at 720p60fps at a bitrate of 3500kbps. I still have to put OBS's process priority to "high" under advanced and set the CPU priority under Output to Superfast and I don't get the encoder overloaded error anymore, though picture quality suffers as a result. Hardware encoder can output at higher settings but leaving OBS at high cpu priority is a must. Doesn't impact in game FPS at all no matter what I set it at. But having chrome or firefox open to watch a preview of the stream isn't happening, every time I load the stream in the browser the video hangs while audio continues, after a short while the stream feed starts buffering and eventually craps out. Also make sure you have that nvidia experience part of the driver package uninstalled, there's reports elsewhere that it actually causes problems.

Looking at resource monitor though I can see that CPU 0 gets pegged at 100% while all the other cores are sitting around 10% to 12%.

 

[2FU4You]

I have the same problem
If I go into a Game then the cpu 0 goes to 100% and the other run 0-15%
And the only because I have open OBS without Stream
When I try to stream then I have extreme performance problems
i try to run my CPU @ 4,6 Mhz with prime 95 over 15h and no problems

 

[Normul8or]

OMG! This is exactly what is happening to me. I just went back and tried what you guys are describing about having the game selected and then tabbing to OBS. When I'm on my game the encoding error happens even though my CPU usage is around 40% total and less then 15% for OBS, then I tab to OBS, away from the game, and the encoding error stops. It cost me an entire broadcast last night, and I have a Noon to Midnight scheduled to start in 3 hours which will not go off without a solution!! As of Saturday night I was streaming flawlessly with no problems. Took Sunday off, then I start up my computer Monday and I'm having major problems without changing a single thing. And now I learn that many people are having the exact same problem in the exact same time frame...well at least I know it's not me!

 

[Fenrir]

So, this sounds like everyone here is not quite sure how games work on Windows, and is lumping general performance issues into this thread.

If you're playing the game fullscreen, and you alt+tab out, the game will stop rendering. This means CPU and GPU usage will be basically nothing from the game itself. When you tab back in, the game starts rendering again, and CPU/GPU usage will spike back up to where it was before.

If you're playing the games windowed or borderless windowed, many modern games have functionality that will significantly lower process priority (or in some cases, limit FPS to 10 or 15) while the game is not in focus.

For reference, I play FFXIV on my PC at 4k resolution, and I have no issues streaming or recording it regardless of what window is in focus, because my PC can handle the load.

Other things that have been known to impact performance in strange ways is the new Windows 10 Creator Update "Game Mode" which I would highly recommend you disable if you have not already. See our Win10CU thread for more information on that.

 

[jakobmiller]

Comment

But I am running a game that is 10 years old. 300 fps. GTX 1070, i7 7700K. Whatever setting I put up, I get encoding overloaded as soon as I select the game window. I didn't get that a week ago. What causes that? I don't have the game mode thingie on my w10.

 

[SailorDeath]

I found a user post on the ffxiv subreddit from over a year ago, someone having the same exact issue, seems to be specific to FFXIV and a few other games, sadly no one was able to help that guy either. It's confusing as to why this would start all of a sudden too. Also, at least for ffxiv, you can turn off the render fewer frames when game is not in focus in that game's settings, I have that option unchecked and if I select another window the high CPU usage stops and everything returns to norma.

Also, last night I spent some time tinkering in task manager and the ability to set preferred CPU cores in it. With resource monitor open I set each of the three high use programs to their own set of cores (OBS to cores 0 and 8, FFXIV to cores 1 and 9 and firefox to 2 and 10) and here's what I discovered, despite setting the programs to use different cores, they all move to using core 0 when FFXIV is made the primary window, I can see the cpu usage on coes 1, 2, 9 and 10 drop to almost 0 and core 0 go up to 100%. As soon as I select another window, core 0's usage drops to about 30% and the other cores' usage goes back up. The issue is clearly one of bottle necking. When I try this with GTA 5, everything stays where they were assigned. I'm thinking this issue is specific to FFXIV and not the system. (They may be other games this happens to as well, I dunno, Berserk and the band of the hawk is another this does this on for me, but other games like Skyrim, Fallout 4 or Resident Evil 7 don't have this problem)

As for performance issues, I also play at 4K and didn't have these problems, I also have a very high end CPU and it should handle the load, it handled it fine when I was running an FX-9590 CPU with 2 R9 290Xs it should have no problem in a Ryzen 7 1800X with a GeForce 1080 TI. It didn't have this issue until Sunday evening as it ran fine on satuday for 6 hours without any problem. Game performance itself is fine, framerate fluctuates between 50 and 70 in high density areas but in areas where I'm alone, like in my house, it holds steady at 65fps, meanwhile OBS and chrome are choking the cpu because when streaming is active CPU 0 is pegged at 100% Turning up OBS's CPU priority in the advanced menu fixes the encoding errors (provided I also set cpu useage priority to super fast or higher)

Here's why I think it's a problem with FFXIV (and some other games) though and not OBS (the problem in OBS is a symptom of what's really going on.) If I close OBS entirely no streaming, and then fire up FFXIV, then I open a stream up in firefox and have it playing, everything still gets loaded on core 0. But 2 programs doesn't tax the cpu as much as 3, so the core doesn't experince any bottle necking, video gets a little stuttery but still plays instead of the video freezing. At this point, I don't think there's anything OBS can do here as it's most likely a problem with the game or with Windows 10 in general. (also I don't have Windows 10 Creator Update installed, it actually asked me if I wanted to install it this morning)

Here's what I did to get it so I can at least stream:

1. Open OBS, go to settings.
2. Select the Output tab
3. I have output mode set to advanced
4. for Encoder I selected NVENC which is a hardware encoder
5. Rate Control is CBR
6. Bitrate is at 3500, my internet connection is 50mbps down 6mbps up, and handles uploading fine but you can adjust this if it's still giving you problems.
7. Select the Video Tab
8. Base canvas resolution: 1920x1080
9. Output scaled resolution: 1920x1080
10. Downscale filter: Bicubic
11. FPS 60
12. Select the Advanced tab
13. Process Priority: High

That's it, I can at least stream on those settings and not experience a dip in game play or in stream quality. I also MUST keep any webpage that's playing video closed as the whole bottle neck issue rears its ugly head video stutters and stops if that's running. I suggest if you have a phone and wifi to install twitch on there and open your stream if you want to have a live view of it. (I do this because I want to know if my connection starts having problems and also to see what my viewers are seeing) This really is a workaround and not a solution and I still don't know what changed on my system to cause this to start happening. In the past something like a windows patch or driver update could do that to you and rolling back would fix the issue, but here there were no patches or driver updates until after this issue started (updated my chipset and video drivers trying to fix the issue) Again, I think this is either a problem with the game, windows or some background process interfering with the way the system distributes work to the CPU.

 

[jakobmiller]

But I want to render on the CPU! Entire reason why I upgraded it :/. I tried streaming with NVENC, but it was stuttering a bit on the vod and I didn't get as good quality as I did last week. Hmm..

 

[jumpinspider]

I'm having the same exact issues today. Everything worked perfectly yesterday and I haven't touched any of the settings. Tried multiple games, no luck.

 

[jakobmiller]

http://imgur.com/949maKa

This is what I am trying to do as shown on the image. Uploaded new log file as well.

 

[jumpinspider]

This is going to sound a little silly, but I think I've found a temporary fix. I was having the same exact problem today. CPU0 was the only one in use when I clicked on the game window. If you actually use alt+tab to cycle to the game, it usually distributes the cpu usage correctly and everything runs fine. On the few occasions CPU0 was still being overloaded, I alt+tabbed back out and back in again and it fixed the issue.

I have no idea why clicking back on the game (using 2 monitors) would cause this issue while alt+tabbing doesn't. The only change made to my pc since yesterday was the Windows 10 update.

 

[Fenrir]

That sounds an awful lot like the Windows 10 CU game mode causing issues. Try disabling game mode and the Fullscreen Optimization options for OBS.

All of this is described in the Windows 10 CU thread I linked.

 

[SailorDeath]

Yeah it sounds like the CU problem but I've checked just to make sure it didn't ninja install, it's not on this system at all. None of the menu mentioned in that thread are on my system

 

[royal198]

I am having same issue out of the blue! Worked fine two days ago and yesterday I am experiencing the same issues like guys above.
I don't believe it has something to do with performance of CPU or any other hardware, it's not simply possible it happened to all of us just out of the blue :X

Any it doesn't matter how low encoder setting I set. It is all the same!

 

[SailorDeath]

Hey Fenrir, I just wanted to let you know, while this fix didn't get listed here or in the Windows CU page, I did see you mention this elsewhere though. It was what was causing all of my issues: Avast Gaming Mode. I normally have it shut off but sometime over the last week an update must have turned it back on. Now I'm no longer having any kind of issues and temporarilt disabling avast while troubleshooting did not fix the issue either. So those of you who are also having this issue, if you're using avast make sure you have gaming mode turned off (you can access that by opening avast, going to "performance" and turning off Game Mode. You can still set it to silent so it doesn't give you those annoying pop ups but it looks like something with the scanner is causing interference with the way OBS operates as well as streaming video. I can now broadcast using the software encode at 60fps 720p with the program set to normal priority and encoding at very fast which was choking before and I can even have a window up now with a live preview of the stream and not have it freezing the moment I put focus back onto the window.

I'm hoping this was your guy's problem as well. I don't know much about the other scanner like AVG but I'm willing to bet they have a similar feature that can be turned off. Thanks, for taking the time to help with this issue and I'm glad I saw that post it fixed everything.