Lag/Stutter Recording/Playback

[Yuki Mori]

Hello!

I would kindly like to ask for help with the following problem. I'm trying to record videos at my native resolution which is 3840x2160. Now, whenever I try this with the following settings in OBS:

Settings->Video:
Base-Canvas-Size: 3840x2160
Scaled Resolution: 3840x2160
Filter: Lanczos

What happens is when the above settings are set to my native resolution either the game starts stuttering during recording Filament or the recorded video does. I tried with Grimrock with the only difference being that there is stuttering in the video but not while recording. I read some guides, I watched videos, and what I got from this was the impression that when I set both of the above settings to my native resolution there should be no stuttering. Not in the game and not in the recorded video.

When I change the scaled resolution to - say: 3072x1728 both lag and stutter during recording and recorded videos is gone. This I fail to understand. When I set my native resolution to something below that - which obviously is not what I want - say 2560x1440 and set the above settings to the same resolution there's no lag or stutter going on.

My output settings are as follows:
Output Mode: Advanced
Type: Standard
Recording format: mkv
Encoder NVIDIA NVENC H.264 (new)

Rate Control: CBR
Bitrate: 300000
Preset: Max Quality
Profile: high
[checked] Psycho Visual Tuning
Max B-frames 2

Advanced Settings:
Renderer D3D11
Color Format: I444
Color Space: 709
Range: Partial

My ultimate question is why can't I record a video with these settings without my system to lag stutter like crazy despite whatever I tried in Windows 10 (Pro) - the NVIDIA Control Panel, etc. when I set both output methods to native resolution? Can I somehow disable any scaling and filter being applied so that OBS simply records at my native resolution and see what it does? I seem to have seen that this should be possible.

What I'm using is OBS Studio 25.0.8 (64 bit) Maybe there is a different version that this issue (whatever it is) doesn't exist? Could you recommend a version I could try?

Please help me to figure this out!

 

[Yuki Mori]

Here is a little update on my problem. I don't want to jinx it but it seems that my problem is resolved. Maybe the following will help someone else having the same issue:

I uninstalled and wiped the remnants off my HDD, than installed 24.0.3 and tested with little success. There were recording stutters and playback issues. Wiped it and installed OBS 25.0.8 (64 bit) again. Still using the same settings as outlined above first, set my monitor to its native resolution, 3840x2160 @60Hz I recorded some demo videos and when playing them back the stutter/lag issue was there. Sometimes more and sometimes less obvious ... I do notice even the smallest lag and it is driving me crazy!

If recording went smoothly even at very high resolution/quality settings and playback is choppy it must have to do with the video player. I used to use VLC, tried to change the output renderer from D3D 11 to OPENGL to OPENGL for Windows and some others. The issue was still there and when looking at the media info it reported 30 to 50 lost frames on average. About the same was the case with Windows Media Player and watching in Pictures. Each had their own issue with the recorded material: some showed excessive tearing, some more or less lag.

Then I found a topic on here that pointed me to PotPlayer which according to the OP was the only one they used that displayed the output smoothly. Yes, playback was a lot less choppy, almost completely smooth. So this reinforced the believe that it must have to do with the players and not with the recording as the logs didn't give away that there were excessive frame drops going on or any errors that occurred.

In another topic on here I found the other part to the solution of my problem. The setting of the monitor must be exactly 60Hz or exactly 120Hz etc. and it will playback and should record smoothly no lag no choppiness. In the advice given the OP mentioned to set a profile (NVIDIA in my case) with the resolution and then set it to 59Hz. I haven't tested what would happen if it is set to exactly 60Hz but will which may improve things even further. So I created a custom profile and set it to 3840x2150 @59Hz, didn't select that profile but went with my native resolution @60Hz.

Another thing I did was to set a max frequency in the NVIDIA control panel to 59Hz as well. This is something I already did when the problem first occurred but did nothing so I guess it doesn't do much now.

After this I did yet another test recording, playback was absolutely smooth no chopping, tearing, lag, stutter! Wonderful! I even tested with following settings in a couple of games, The Legend of Grimrock and Filament mostly but also Operencia. Used even higher quality settings which are:

Output Mode: Advanced

Encoder: NVIDIA NVENC H.264 (new)
Rate Control: CQP
CQP Level: 3
Keyframe Interval: 0
Preset: Max Quality
Profile: high
[checked] Look-ahead have to try with this setting off as this sometimes messes around with recordings not very often though but ...
[checked] Psycho Visual Tuning

Video:
Base Resolution: 3840x2160
Output Resolution: 3840x2160
Downscale Filter: Bilinear (may change this to Lanczos or Bicubic later to see what it does)
Integer FPS Value: 60

Advanced-Video:
Renderer: Direct3D 11
Color Format: I444
Color Space: 709
Color Range: Partial (must test with full though there seems to be a gamma issue when using this?)

Filament The quality in the following videos is very low. It didn't impact the lag/choppiness issue (high/max quality or low/very low)
Before: Before Video
After: After Video

Filament with qpr 8:
After: Smooth Playback

The Legend of Grimrock This was recorded with qpr 3 so higher quality but because its not matching YT the quality isn't what it could be. Doesn't matter, it was testing for lag and the recorded scene here, with the settings mentioned above - I couldn't even play it back without lag, stutter, lag, lag at every turn ...

After: Performance Smooth

System Specs:
Asus Prime Z-390-A Gaming
I7 9700 @4.9GHz
ASUS NVIDIA Strix 1070
Vengeance LPX @3200 MHz DDR4
2x SSD (1TB, 2TB)
1x External Drive for recording (4.0TB WD drive)
OS: Windows 10 Build 10.0.019041
OS-1: Linux (Slackware [In Slackware I trust :-p])

I hope some of this information does help someone else solve their problem. Also there is a small request: Where can I learn about custom muxer settings? Presets are great - but - being a curious person I do want to learn more about this and how it can improve recordings. So, if you have some FOR ABSOLUTE DUMMIES (which is me) resource let me know!