Resource icon

How to: Troubleshoot high FPS recordings (120, 144, 240, 300 FPS recording)

A collection of troubleshooting advice snippets to help you record at 120, 144, 240 and higher frame rates, assembled by me, Nakajima Megumi#7432. I'm operating under the assumption that your machine is already capable of recording at 60 FPS - if not, get that fixed first with the help from OBS Studio Discord server or any other means of your choosing, then you can try to push for a higher frame rate. Do not skip this part.

Grab MSI Afterburner and RivaTuner Statistics Server from https://www.msi.com/Landing/afterburner (installer includes both) if you need to collect performance statistics.

Contents:
-2. "Minimum system requirements" and hardware encoding throughput
-1. Common mistakes and things to avoid
0. How to access frame rates in excess of 60
1. NVENC Start Failure
2. NVENC crash (stuck "Stopping recording...")
3. Encoding lag causes and symptoms
4. Mixed refresh rates
5. Bad frame pacing, bad game performance
6. Non-integer refresh rates / FPS limits
7. Corrupt video output
9. I'm using Custom FFmpeg output



-2. "Minimum system requirements" and hardware encoding throughput

Hardware encoders are not infinitely fast. Unfortunately, neither AMD nor NVIDIA provide specific information about their throughput, and that throughput may also change depending on what else you're doing on the computer; here's some information I was able to collect:

- Turing NVENC (GTX 1600 series except GTX 1650, RTX 2000 series, RTX 3000 series) should be capable of ~630 FPS at 1080p. (Tested with someone else's GTX 1660)
- Update 03 Dec 2021: Looks like someone had issues with the "Quality" preset with a 2060 Super at 1080p 420FPS, changing preset to "Performance" and disabling "Psycho visual tuning" fixed this.
- Pascal NVENC should be capable of ~260 FPS at 1080p, per chip. Meaning, a GTX 1050/1060 should yield 260, but a 1080/1080Ti should yield double that, ~520 FPS, as they have two NVENC chips. (Tested using my own hardware as well as some other people's hardware)
- The AMD encoder appears to be capable of 240 FPS recording on an RX 480. (Someone in the obs-studio Discord managed to get it to work. No idea how.)



-1. Common mistakes and other things to avoid

Do not use the "Max Quality" NVENC preset

On Pascal and older NVENC chips, this may improve quality, but NVENC usage and GPU usage will at be least doubled. Very risky, and almost no quality gain on Turing and Ampere cards. Use "Quality" or "Performance".

Do not use excessively low CQP values

H.264 is very inefficient at high frame rates; using CQP may result in significantly higher bit rates than what you're used to. Start with a high QP value (like 30) or VBR, then adjust your settings when you make sure that your storage device can keep up and your recordings don't come out corrupted.

Do not write to network disks or external (USB, etc) storage

Just don't do that, write to local disks. Even if the storage device itself is fast, USB or networking overhead might still give you headaches. Slow storage may erroneously show up as "encoding lag", or corrupt your videos without any error messages.

Do not use excessively long keyframe intervals

While this shouldn't break the recording, your video editing performance may suffer greatly with certain video editing applications. Using a value of 0 will use a keyframe interval of 250 frames, this is preferred if you're recording at more than 250 FPS.



0. How to access frame rates in excess of 60

In Settings > Video, set FPS type to Fractional. If you want a flat 120 FPS, enter 120 and 1 (120 divided by 1 equals 120). If any of your devices are 29.97 fps or 59.94 fps, such as webcams, you could get slightly better frame pacing for them by multiplying your desired frame rate by 1000 and entering 1001 in the denominator field. For example, if you want 119.880 FPS instead of 120.000, enter 120000 as the numerator and 1001 as the denominator.



1. NVENC Start Failure

failed to start recording.png


Open your current log file (Help > Log Files > View Current Log > Open file) and search for "NV_ENC_ERR".

NV_ENC_ERR_OUT_OF_MEMORY has 3 possible causes. Unfortunately, NVIDIA chose to throw them under the same error code.
Option 1: you already have 3 NVENC instances running. NVIDIA imposes an artificial limitation where trying to start a 4th video encoding instance will not work, even if the hardware is capable of doing that. Stop other applications that are currently using NVENC, such as GeForce Experience.
Option 2: your game claims all available VRAM. This will NOT show up in Task Manager or other tools that report VRAM usage. With some games, you must start all your NVENC outputs before starting the game, you won't be able to start any new NVENC instances later. (Seems to be a frequent occurrence with these games: Half-Life: Alyx, Realm of the Mad God Exalt, rarely Call of Duty: Modern Warfare (2019))
Option 3: You're actually out of VRAM. Extremely unlikely, but check Windows' Task Manager anyways. You need enough free physical VRAM, virtual/"shared" VRAM won't help.

NV_ENC_ERR_INVALID_PARAM means your current encoding settings are bad. The only way to pull this off with OBS Studio without plugins would be to use an invalid resolution (i.e. width or height greater than 4096 when using most H264 encoders), but the StreamFX plugin unlocks additional options that also make it possible to use invalid settings. (I think this error might also pop up if your NVIDIA driver is too old.)

NV_ENC_ERR_INVALID_VERSION means your NVIDIA graphics card drivers are too old. Some OBS Studio updates change the minimum required NVIDIA driver version, so updating OBS Studio may additionally require updating your graphics card drivers.

If you have a different issue, let me know and I'll add it here.



2. NVENC crash (stuck "Stopping recording...")

There is a known issue with recording at more than 60 FPS: if you're using multiple audio tracks, there's a high chance your recording will fail to start - when you stop recording, it'll get stuck "Stopping recording..." and your recording file will be empty. This likely won't be fixed in the foreseeable future. There are two workarounds available:
a) Verify that a file has indeed been created and is growing in size after you start recording. (Select the file in Explorer and press F5 to update its size.)
b) Send all your audio tracks to a secondary OBS Studio instance that will record at a much lower frame rate. You can use the obs-ndi plugin for that if you choose to go that route, I wrote a guide on doing that: https://obsproject.com/forum/resour...6-audio-tracks-with-obs-studio-uses-ndi.1387/

If you're not recording multiple audio tracks and it still crashes, that will be more difficult to troubleshoot:

First thing to try here would be to underclock your GPU core by 400 MHz with MSI Afterburner. (This is safe and will not void warranty.) When you run Afterburner for the first time, you'll be greeted with its awfully confusing UI; locate the cog icon
shit ui.png

, switch to the User Interface tab (scroll the tabs bar all the way to the right) and select a more usable theme, such as Default MSI Afterburner V3.

Slide "Core Clock" all the way to the left (-400 MHz) and hit Apply. If nothing happens and the value unsets itself to +0, restart Afterburner as administrator and do it again.

afterburner -400mhz.png


Try recording your game again. If this helps and your recordings save correctly, you can try increasing core clock to a higher value (still has to be +0 or less, otherwise it may result in overclocking, which voids warranty).

Unfortunately, this is the only troubleshooting option I can provide for this issue; some reported that it doesn't help them, and I have no clue whatsoever what else may be going wrong here. Falling back to generic troubleshooting steps, you could try running DDU https://obsproject.com/forum/resources/performing-a-clean-gpu-driver-installation.65/



3. Encoding lag causes and symptoms

Generic troubleshooting steps still fully apply: OBS Studio must be running with administrator privileges to enable the GPU priority fix, and your peak GPU usage must not exceed 90%. Afterburner can help you with that - enable the GPU usage graph and press "Detach", you can see the highest recorded GPU usage. The moment your GPU usage shoots above 90%, you might get stutters in your recordings.
afterburner max gpu usage.png


Verify that your game's frame rate is being capped at your recording resolution; ideally, you should never dip under your target frame rate, although realistically most games aren't that well optimized and will likely stutter and dip under your target frame rate. Use NVIDIA and AMD driver settings to cap the frame rates; some games come with their own FPS limiters, but those typically favor lower latency over better frame pacing, which may be bad if you're trying to make smoother recordings. If you intend on recording at 120 FPS, cap your game at 120 until you start getting good 120 FPS recordings; capping at 240 could be introducing issues.


If you're still having encoding lag with NVENC, try this set of mostly fail-safe settings, and then adjust as you go:

safe settings.png


A few notes regarding the choices made:
"Rescale output" will impact performance and/or quality if enabled.
We shouldn't need custom muxer settings unless you're also doing fragmented MP4 or something else like this.
CQP is always preferred for higher video quality and slightly better encoder performance, but VBR at 35 Mbps at least guarantees that I/O performance won't be an issue. Change this when you're done troubleshooting.
"Max Bitrate" refers to VBV buffer size.
A keyframe interval of "0" will use 250 frames. (Longer interval means better compression, but may reduce performance in video editors and video players.)
"Performance" preset, compared to the "Quality" preset, improves hardware encoder throughput, at the cost of quality. In a "normal" (60fps) recording scenario, this is hurting quality for no benefit, but here, we might really need all the throughput we can get. Change to "Quality" if you can get away with it.
Profile: high. I don't know if the other settings will make things better or worse. Note that high444p will automatically be used if you choose I444 in Settings>Advanced.
Psycho Visual Tuning, B-frames: both disabled. The performance footprint will likely be too high. Just use lower QP values if you want better quality.

Note that I'm recommending flv and only a single audio track; OBS Studio uses slightly more disk I/O with the mkv muxer, and some people reported having issues with it on slow storage. If you don't need multiple audio tracks, you can keep using the flv muxer; it's both crash-safe and less prone to break at high frame rates. Check the beginning of section 2 if your MKV recordings occasionally come out broken.



4. Mixed refresh rates

If you're using Windows 10 release 2004 or newer, skip section 4, this issue is fixed. (Hearsay: if the difference between the lowest refresh rate and the highest refresh rate of your displays is greater than 3x, i.e. the lowest is 60 and the highest is higher than 180, you could still get issues. Ping Nakajima Megumi#7432 in the OBS Studio Discord if you have more info on that.)

Using multiple displays with different refresh rates, such as 60 and 144, will cause severe stuttering on all displays as soon as you run any hardware-accelerated application on any low refresh rate display.
If you're on Windows 7, you can disable DWM by switching to a "classic" (non-Aero) Windows visual style, or by ticking the checkbox in Settings>Video in OBS Studio.
If you're running Windows 10 release 1909 or older, or Windows 8 or 8.1, or don't want to disable Aero on Windows 7, you have a few options:
a) Unplug all low refresh rate displays (if there are any) and bring all your high refresh rate displays to the same refresh rate (like 120).
b) Disable DWM on Windows 8 or 10 (use your favourite search engine to find out how).



5. Bad frame pacing, bad game performance

Many games just aren't optimized well enough to run at 120 FPS or higher at all times. A few troubleshooting steps may be taken:

- Start recording before starting your game. This ensures that the GPU stays in a high power state the entire time, and that enough VRAM necessary for hardware encoding is allocated.
- Enable the frame time graph in MSI Afterburner and watch it. (Watching it isn't going to make your frame pacing any better, but it might give you an idea of how good or bad it is.)
- Use a third party frame rate limiter. (In this context, "third party" means "not built into the game", i.e. NVIDIA control panel, or RivaTuner Statistics Server.) This will increase input latency by at least 1 frame, which is bad for competitive titles, but also improve perceived smoothness and reduce the likelihood and severity of visible stutters, both for you and for the video output.
- Lower your frame rate limit. Ideally, lower it all the way to your target frame rate - i.e. if you want to record at 120, don't cap at 240, cap at 120. Or, if your end goal is motion interpolation, lower it to the lowest frame rate that still benefits you.
- Lower the graphics settings. Most games don't explain which settings affect rendering (GPU) performance and which settings affect simulation (CPU) performance, so that may be difficult to figure out. Again, make sure your peak (max) GPU usage is no higher than 90% at your usual clock speed (since graphics cards downclock themselves when at low load, MSI Afterburner may erroneously display a very high peak that happened when the GPU switched power states, i.e. when launching a game; reset Afterburner's stats after you launch your game); if you peak above 90%, you risk having issues.
- Pin the game to specific cores in Task Manager. On Ryzen systems, try the first half, or the second half. "Disabling" SMT/HT by unchecking every odd core may also help on CPUs that have these features. Most modern games will see no benefit from this, but some older games might have better frame pacing. The same can be done to OBS Studio, and any other applications eating CPU time in the background. If you'd like to automate this, check out Bitsum Process Lasso https://bitsum.com/
- Make sure the CPU power saving mode is set to "Ryzen High Performance" (if available) or "High performance". Don't touch GPU power saving mode, leave it at "Optimal power" or the AMD equivament.
- Close other applications that eat CPU time. Games generally don't like having to fight over CPU time.
- Disable preview in OBS Studio for a (very) slight reduction in GPU usage. (You can set a hotkey to toggle preview in Settings > Hotkeys.)
- Disable Microsoft Windows Defender. Its performance footprint doesn't accurately show up in Task Manager, and it is quite severe. (More info: https://www.youtube.com/watch?v=pzeKPiwEfXk&t=236s )



6. Non-integer refresh rates / FPS limits

If you have a refresh rate of 75, 100, 144, 165, 200 or any other "oddball" refresh rate that doesn't evenly divide by 60, dial it down to the nearest exact multiple - 120, 180, 240, 300, 360. Do the same to your frame rate limiter, too. Not doing so will hurt frame pacing, making it perceptually less smooth than it might have been otherwise.



7. Corrupt video output

Use mpv or SMplayer to verify the issue really is with the video. Not all video players are capable of playing high frame rate video properly.
Apart from that, the most common cause of corrupt video output seems to be excessively high video bit rate, check section -1.



9. I'm using Custom FFmpeg output

Do us both a favour and switch back to "regular" Advanced output mode, FFmpeg output is exponentially more difficult to troubleshoot. If you want lossless audio, use Audacity. If you need NVENC HEVC, use the StreamFX plugin (though it should be noted that HEVC is more expensive to encode with NVENC, expect less throughput than H264).

I don't know what happened to section 8. Maybe I can't count to 8?
Last major update: 29 Mar 2021
Last formatting update: 27 Dec 2021
Top