Question / Help Encoder Overloaded - What am I doing so wrong?!

[Juliannb]

TL;DR: OBS randomly decides to say "Encoder Overloaded", ruining my recordings, and I have no idea what am I doing so wrong to let this happen. Help, please?

Good day, everybody!
I want to start this post by saying I love OBS Studio's freedom, it's easily one of the best recording tools I have the pleasure to use... and also the displeasure, because of how unreliable this tool is becoming to me.
My build is as follows:

• OS: Windows 10 Pro x64;
• CPU: Intel i5 7600K 3.80GHz;
• GPU: ASUS Radeon Rx 580 8GB;
• Motherboard: Gigabyte H270-Gaming 3;
• RAM: Corsair Vengeance 16GB (2x8GB) 3200MHz;
• Storage: Corsair MP500 M.2 240GB + WD 3TB 7200RPM.

A more than capable build I have here, right? Now, here's the latest Log File and a screen shot of my Game Capture settings and Recording settings (just in case), so you have a general idea of what I have:

• Log: https://pastebin.com/McBmT3uX
• Game Capture settings: https://imgur.com/jjtxvcu
• Advance Recording settings: https://imgur.com/34kmwQu

(I know I can post pictures here, but this is a copy-paste from another forum)

I've been using OBS for the past year and the results varied from time to time, but it has behaved according to my expectations. That was until the last month, with the "Encoder Overloaded" message that pops up randomly, even when there's a blank screen being recorded. This is happening more often than not, making it very unreliable, to the point of making me find alternatives (which I can't find).
I've watched tutorials, read posts and even asked people who use this tool on their daily basis, but I just cannot understand what am I doing wrong, I randomly get the message "Encoder Overloaded" on games like Castle Crashers and Axiom Verge (I'm not joking! And don't even get me started on Dead by Daylight).
Let's take these 3 games plus an extra game I didn't mention yet as examples:

• Castle Crashers: A game that uses only 2D sprites, no fog effects, no 3D models, no particle effects or anything that could or should overload anything. Every single time the game needs to load, encoder overloaded. I could be playing Crysis while Castle Crashers is running and both would be playing alright (despite the fact of being able to);
• Axiom Verge: A 16-bit looking platformer. Small sprites scaled up, nothing overkill, right? After 40 minutes recording, encoder overloaded. I managed to (once) record Getting Over It for an hour and a half and no stutter nor overload;
• Dead by Daylight: A 3D horror survival game, this game is a mess. I'm starting off by saying this game is poorly optimized and I hardly see it'll get the proper optimizing it's begging for. The only way I can record this game without encoder overloaded is by playing the game at Medium, and even then, when the doctor is around, I get encoder overloaded, I have to play the game on Low (which doesn't look too bad, it's actually brighter).
• Spageht: Now what the hell is this game? It's a horror game in a somewhat similar fashion as The Boogeyman. The game uses 3D assets and it's not incredibly detailed with the graphics, but alas, it plays just fine, but it manages to give me the encoder overloaded message the moment I start playing.

So far, the possible solutions I've found are these:

• Disable antivirus while recording. Didn't work;
• Disable Windows Defender. Already done that, didn't work;
• Disable Firewall. What does it have to do with anything?! No, it didn't work;
• Lower the recording quality. As much as I hate doing that, I did, and it didn't work either;
• Record using x264 encoder. No, I already give the CPU enough stress with the highly-demanding games. Also, I can't reach the same level of quality without over-killing it, that's why I decided to go with H264 (and if that's not true, then prove me wrong);
• Record using H265 encoder. Didn't work;
• Don't record those games. Yes, I've been told that. Very funny;
• Use Game Capture instead of Display Capture and don't have them both on the same scene. Done that, didn't work;
• Reinstall the OS. Done that already, as tedious as it sounds, but didn't work;
• Change OBS's Process Priority to a higher one. Didn't work;
• Change the game's priority to below normal. It sort of works, I haven't got the issue with Dead by Daylight (the only one I tested it on), but I refuse to accept this as the ultimate solution, as no other tool I've used require doing this. Not to mention, my computer isn't a paperweight;
• Upgrade my PC. Seriously? Have you seen the beast I have under my desk? (no sexual euphemism intended);
• Use another recording tool. Haha, good one. I tried Bandicam with H264 encoding, works wonders (except for Dead by Daylight), I tried ReLive and it works perfectly with every game I play, but they don't cover my necessities. I would move if there was one that lets me record a video at any resolution I want (I record at 1920x1800), lets me record with more than 2 audio tracks without the need of any complicated setup and record my webcam separately (hence the resolution), all at once. No, XSplit is not an option;
• I've noticed a lot of people, if not, EVERYBODY use Nvidia GPUs instead of AMD. I've been told it should work the same, but seriously, nobody, I repeat, NOBODY uses AMD GPUs, all of them mention Nvidia. I'm not switching to Nvidia, not for recording's sake at least;
• What I haven't tried yet is to record on a separate SSD unit, but it begs the question: do I really need to reach to that point to record with OBS? I know I'm asking to record at 35k bitrate, but COME ON! I can record with other tools at that bitrate no problem! I'll do the test once I'm able to (I have a spare SSD on my other PC, I just need the time);
• I haven't tried recording on Linux with OBS yet, but nobody mentioned it at all. I may try it one day, no promises though;

This sounds more like a rant than anything, but I'm seriously losing my mind over this, recording gameplay footage should not be this difficult! Heck, EposVox records his footage at 250k bitrate! Sure, he uses an SSD to record that, but how is that I can't reach one eighth of quality without issues?!
I don't want to stop working on videos, as I have a lot of fun making them and it's also a good income, nor I want to stop using OBS for my daily basis, but by all means, I need help, I really don't know what I'm doing so wrong to get these random encoder overloaded warnings.

I see you made it through the end of this long post, good for you! If you have any tips or suggestion that is not listed here and may fix this issue, please, share it in the comments, I'll make sure to try it out once I get the chance and respond with the results.

 

[koala]

Unfortunately, your log contains only a 3 second recording. For these 3 seconds, there are no errors, no problems to see. Please record a few minutes (3-5) of the game you have trouble with and post a log that contains this recording attempt.

If if comes to AMD versus Nvidia, an AMD card is as capable at displaying games as Nvidia cards are. They are not per se incapable to support recording. What's a bit inferior on them in comparison to Nvidia is the hardware encoder. Nvenc quality is almost the same as x264 on veryfast preset, which is the OBS default. AMD VCE (which are you using) isn't that good. I saw recordings with this, they always contain some grain, regardless how high the bitrate you push.

Your CPU has an onboard GPU capable of Quicksync. For recording, the Quicksync encoder is as good as Nvenc, if you tune the settings accordingly. Your mainboard supports the iGPU, so go into the Bios of your machine and enable the iGPU. You don't have to plug a monitor in the mainboard outlet - just enable iGPU in Bios. Windows will autodetect it and auto-download the current Intel drivers for it. After that, if you find "Intel(R) HD Graphics" in device manager of Windows, start OBS. You will find that you now have "Quicksync H.264" as additional encoder. Try this instead of the AMD encoder. Use ICQ as rate control instead of CBR and a ICQ quality value between 20 (better quality, larger files) and 23 (not so good quality, smaller files).

 

[Juliannb]

I apologize for the lateness of my reply, but I got you some news:

Unfortunately, your log contains only a 3 second recording. For these 3 seconds, there are no errors, no problems to see. Please record a few minutes (3-5) of the game you have trouble with and post a log that contains this recording attempt.

My mistake. I posted the wrong log file, but reading what else you said, I don't think it's needed anymore.

If if comes to AMD versus Nvidia, an AMD card is as capable at displaying games as Nvidia cards are. They are not per se incapable to support recording. What's a bit inferior on them in comparison to Nvidia is the hardware encoder. Nvenc quality is almost the same as x264 on veryfast preset, which is the OBS default. AMD VCE (which are you using) isn't that good. I saw recordings with this, they always contain some grain, regardless how high the bitrate you push.

That is very sad to hear, because I trusted AMD/ATI ever since I got my first ATI Radeon HD 5750 up until this point. After knowing this... I might as well stop buying AMD graphic cards, since I can't fully trust them on the recording side. ReLive works perfectly, but I'm very limited with the options.

Your CPU has an onboard GPU capable of Quicksync. For recording, the Quicksync encoder is as good as Nvenc, if you tune the settings accordingly. Your mainboard supports the iGPU, so go into the Bios of your machine and enable the iGPU. You don't have to plug a monitor in the mainboard outlet - just enable iGPU in Bios. Windows will autodetect it and auto-download the current Intel drivers for it. After that, if you find "Intel(R) HD Graphics" in device manager of Windows, start OBS. You will find that you now have "Quicksync H.264" as additional encoder. Try this instead of the AMD encoder. Use ICQ as rate control instead of CBR and a ICQ quality value between 20 (better quality, larger files) and 23 (not so good quality, smaller files).

I cannot believe I didn't see that before! I had the iGPU enabled, but never installed the drivers, nor Windows Update download the updates needed to install it... I checked out the optional updates, I downloaded them, restarted and checked OBS's encoders and there it was! Now, I was very worried to use it, since it I thought it'd be using the CPU as part of the process as well, but not at all, it does use the iGPU for the encoding.

• I've tried Dead by Daylight on Ultra and no overloading at all! I'm absolutely shocked;
• I've tried Castle Crashers and... this game is a rare piece, the encoder would overload on the loading screens again, so what I decided to do was running the game windowed, successfully avoiding the overload. I still don't understand the reason behind this behavior, but I'll let it slide for now;
• I've tried Axiom Verge for 1 hour straight, no overloading at all either!
• I haven't tried Spageht yet, but seeing the improvement here, I have the feeling it'll work.

So what it boils down to is the inefficiency of AMD's encoder... I'm glad I managed to fix this by using my iGPU, but this is really aggravating and I'm very disappointed by... what am I exactly supposed to be disappointed at? AMD's encoder or OBS's way to handle it? Since ReLive handles it without any issues, it's hard to tell.
Why am I worrying about this, you may ask. I have a friend of mine with a similar build, but the main difference is the CPU (he has a Ryzen 7 2700x), and he's having the same difficulties with recordings as I do. As far as I know, the only solutions are using ReLive, changing to Nvidia GPU or changing motherboard and CPU, which the thought alone is outrageous due the money he has to spend. Is there really nothing else he can do?

Anyways, thank you very much for your reply, it has been of great help!

 

[koala]

He should try to limit the GPU usage to 90% to give OBS and the AMD encoder room for processing. Limit the fps by enabling vsync or any ingame limiter.

 

[Catalin Radu]

Same exact issues. I have Ryzen 1700x with RX 480 8GB VRAm and 16 RAM, recording on a SSD HyperX Fury

No matter what settings i use, and I have tried all combinations, the video freezes and has huge lag.

No issues on AMD ReLive, where I can record and game at Ultra settings on games like AC Odyssey. As we speak I am troubleshooting, looking for answers. Most of the times I get Encoding overload. No matter the settings it's always the same.

You can find my log attached

 

[Catalin Radu]

Please anyone?

 

[Narcogen]

You are not getting encoding lag, you're getting rendering lag. Completely different problem.

12:19:06.388: Output 'adv_file_output': Number of lagged frames due to rendering lag/stalls: 18004 (50.8%)
12:19:06.389: ==== Recording Stop ================================================
12:19:06.389: Video stopped, number of skipped frames due to encoding lag: 310/35409 (0.9%)

12:37:10.738: Output 'adv_file_output': Number of lagged frames due to rendering lag/stalls: 29255 (48.6%)
12:37:10.738: ==== Recording Stop ================================================
12:37:10.738: Video stopped, number of skipped frames due to encoding lag: 46/60197 (0.1%)


You need to cap your in-game framerate. OBS does not work the same way as AMD ReLive or Nvidia ShadowPlay. OBS needs to render frames prior to encoding and needs GPU resources to do this. If your game maxes out your GPU, frames do not get rendered, and the above log message indicates that half the time, OBS cannot render a frame, which means half of the frames in your file are duplicates.

13:05:13.344: Output 'adv_file_output': Number of lagged frames due to rendering lag/stalls: 47927 (50.9%)
13:05:13.345: ==== Recording Stop ================================================
13:05:33.478: Video stopped, number of skipped frames due to encoding lag: 73612/81654 (90.2%)

This attempt includes both rendering lag AND encoding lag, but since rendering happens first it's not that important.


I would start a fresh profile and scene collection and run the Auto-Configuration Wizard from the Tools menu to develop some sane defaults for your machine, and then adjust as required.

 

[Catalin Radu]

Thank you @Narcogen

I had tine to take out my RX 480 and use a GTX 1050Ti. With the 1050 i did not have any recording issues. The problem being that the 1050 is less powerful than my RX so the games did not work that great. But the recording settings were the same.

So how can a more powerful graphics card record worse in OBS than a lesser one? Maybe OBS does not play nice with the RX400 series?

I tried Riva Tunner to cap the frames, did not improve the recording. Also the Wizard from te Tools section doe snot really do anything just kinda sets stuff to default.
I am stuck again. Anything that I can improve?

 

[koala]

With the RX 480, you were using the AMD hardware encoder. This encoder uses GPU computing resources for its work. If the GPU is fully loaded by rendering a game, it seems the encoder gets not enough computing power, which results in massive encoding lag. This is not finally proved, but this behavior can be deducted from what can be seen in many log files.
With the 1050 Ti, you either used x264 (software) as encoder or nvenc (hardware on Nvidia). Either don't need computing resources on the GPU, because x264 runs on the CPU and nvenc is a dedicated circuit on the GPU chip that runs on its own. So you did not get encoding lag from high GPU load with the 1050 Ti.
What you can do: With the RX 480, don't use the AMD hardware encoder but x264. Or reduce the graphics complexity and the fps of the game even more until you don't get encoding lags with the AMD hardware encoder. This might result in so low settings you cannot stand, but you can always switch to x264 and not reduce the graphics quality of the game.

 

[Catalin Radu]

Thank you @koala for the idea.

Believe me I have tried. To use the x264 and all sorts of combinations that I found on the internet etc. If there is a magical combo that I can go for it would surprise me in a good way. However I did so many of them,and nothing works.

What is more interesting is that my friend has a RX 570, not as strong as my 480(but that is not important). With his 570 and default recording settings like x264 he can record with no issues. I did not mention this before, however this is very important.
Something in OBS coding, doe snot play nice with the RX 480....don't know if it's just the 480 or the 470,460 etc....(cuz i do not have those cards to test).

New windows installation, new drivers, stock default settings to record etc...i tried a lot of things. Maybe I am missing the magic combination, but it seams to me that without so much effort other graphics cards record flawless while I can not.
_____________

I can perform more tests, give you more logs from OBS. Ask and I will make and provide. This is my issue and I need to get this fixed.

A fast result is to sell this card and buy something different, even so...I need to add more money and right now I am short on that.
In the meantime I record with AMD Relive, and the option to have separate audio track(for my mic voice) while the game and voip chat(discord mic) on the same track. This is very bad, because it is very hard to separate the sound and if you try to edit, both the voice and game sound get edited at once causing unpleasant editing.

The ONLY single possible reason I need OBS is because it has many sound tracks possibility. I can split the audio easy and edit it this way. Other than the above AMD Relive is an excellent recording software with many useful settings.

 

[koala]

Make sure Windows Game Mode is deactivated. Game Mode has been very aggressive in the last Windows update(s) to boost a running game, so OBS doesn't even get a small share of the GPU it depends on.

Other than that, if it's only the audio, as workaround use OBS only as audio recorder and use Relive for video. Record a black screen in 320x200 canvas resolution with OBS and use x264 with crf rate control with CRF=40, so you essentially don't get any video data, but all audio. To be able to sync the audio track from Relive and from OBS, produce a clapping sound at the start of the recording (remember the clapperboard in movie productions? Historically, it's for exactly that: syncing the audio of multiple microphones).
In your postprocessing software, load the Relive video and import the audio from the OBS video and use the Relive audio only for syncing the audio tracks.

 

[Catalin Radu]

same stuff. I turned off the Game Mode in Windows, you were right, it was ON. No difference. I have attached 2 more logs.

I have even went and enabled OC mode on my MSi card, maybe some boost in performance ...i dont know.

But yeah, I will use OBS just for the audio part and will do what you told me. Thank you :)

 

[Narcogen]

20:07:53.223: Output 'simple_file_output': Number of lagged frames due to rendering lag/stalls: 2955 (9.4%)
20:07:53.223: ==== Recording Stop ================================================
20:07:53.223: Video stopped, number of skipped frames due to encoding lag: 11365/31272 (36.3%)

Rendering lag is down from 50% to 9%.
Encoding lag is down from 90% to 36%.

I'm not familiar enough with the AMF encoder, but my understanding is that unlike the NVENC encoder, the AMF encoder *does* use GPU resources. So the rendering lag means the GPU is already overloaded, and then OBS asks it to encode, also.

 

[Catalin Radu]

@Narcogen. That makes sense. So this behavior will happen for all type of GPUS that use the AMF. So everyone out there using OBS with an AMD card should have same issues of encoding, however this is not the case. Else we will have a massive amount of users complaining, but this sis not the case.
Also, performance is not an issue, because there is lots of people with AMD cards lower performance than mine that do not report this.
There must be something wrong with the RX 480.

I never had an Nvidia graphics card, all these years it was AMD for me. But i think it is time to change the team. AMD does a lot of things right, and as for ReLive is an amazing recorder software but I fell like it is time to change. Not ready to jump yet on the expensive RTX 2060 or more. Maybe next year when Dragon Age 4 will come out.


P.S.: Thank you for the Wiki page but i disabled Windows mode already.

 

[Narcogen]

I think you misunderstand me.

What is different between the NVENC encoder and the AMF encoder is that the Nvidia card uses dedicated hardware just for encoding that has no other purpose. So the AMF encoder uses resources on the card that could otherwise be used for rendering your application; the encoder has overhead on AMD cards that NVENC does not on Nvidia cards.

This does not mean that everyone using OBS with AMD will have encoding overload. That does not follow. What it means is that they have more overhead. If they cap their framerate and keep rendering and encoding settings within the specifications their machine can handle, they will not have overload.

If what you are saying is that you do not wish to make any modifications whatsoever to the way you run your applications in order to get high quality recordings, then Nvidia will be a better choice. I have both an AMD RX 580 and an older Nvidia GTX 708, and I am also considering getting an RTX card in the future, mostly for access to the hardware encoder.