NVIDIA NVENC H.264 (new) Causes Corrupt MKV Files Every Time

[MarioGuy789]

Hello everyone, this is my first post on this forum, so please try and be easy on me.

I've been using OBS for quite a while now, but I keep running into the same problem every time I try and record using NVidia's NVENC H.264 encoder. Literally all of the recording files which I have produced with this encoder have come back corrupted. Even after remuxing, which is sometimes unsuccessful, the video output is completely broken, with graphical noise everywhere. I have a perfectly capable system and graphics card (RTX 2070) for running this encoder, but no matter what settings I use, the result is almost always an entirely corrupted MKV file or a retrievable mp4 file in which all of the video is broken beyond usability. The only solution I have is to use the standard x264, but that takes up a lot of resources on my CPU, and when trying to record at the best possible quality, causes subtle, but noticeable slowdown in performance. It would be great if I could put my 2070 to use, but for some reason I simply can't. Either I'm missing something or the encoder is busted for me because I literally can't explain why it wouldn't work.

If anyone has any solutions to this problem or any feedback, I'd be glad to hear it.

Thanks everyone.

(Log file: https://obsproject.com/logs/ZIM7PPM7rv5hkUtF )

 

[rockbottom]

According to that log, you're using x264. Try recording with just 1 audio track.
16:23:15.469: [x264 encoder: 'recording_h264'] preset: veryfast
16:23:15.469: [x264 encoder: 'recording_h264'] profile: high
16:23:15.491: [x264 encoder: 'recording_h264'] settings:
16:23:15.491: rate_control: CBR
16:23:15.491: bitrate: 12000
16:23:15.491: buffer size: 12000
16:23:15.491: crf: 0
16:23:15.491: fps_num: 60
16:23:15.491: fps_den: 1
16:23:15.491: width: 1920
16:23:15.491: height: 1080
16:23:15.491: keyint: 250
16:23:15.491:
16:23:15.540: ---------------------------------
16:23:15.540: [FFmpeg aac encoder: 'Track1'] bitrate: 160, channels: 2, channel_layout: 3
16:23:15.540:
16:23:15.591: ---------------------------------
16:23:15.591: [FFmpeg aac encoder: 'Track2'] bitrate: 160, channels: 2, channel_layout: 3
16:23:15.591:
16:23:15.592: ---------------------------------
16:23:15.592: [FFmpeg aac encoder: 'Track3'] bitrate: 160, channels: 2, channel_layout: 3
16:23:15.592:
16:23:15.593: ---------------------------------
16:23:15.593: [FFmpeg aac encoder: 'Track4'] bitrate: 160, channels: 2, channel_layout: 3
16:23:15.593:
16:23:15.593: ---------------------------------
16:23:15.594: [FFmpeg aac encoder: 'Track5'] bitrate: 160, channels: 2, channel_layout: 3
16:23:15.594:
16:23:15.594: ---------------------------------
16:23:15.595: [FFmpeg aac encoder: 'Track6'] bitrate: 160, channels: 2, channel_layout: 3

 

[rockbottom]

I don't do any multi-track audio recordings but this may help.
6:17:35.904: [CoreAudio encoder]: CoreAudio AAC encoder not installed on the system or couldn't be loaded

OBS-Studio: Enable CoreAudio AAC encoder (Windows)

First of all, if you have iTunes or QuickTime installed, chances are you are already using the CoreAudio AAC encoder in OBS-Studio. But if you normally do not use or install any of those, there is another way to enable it. All we need is the...

obsproject.com

 

[MarioGuy789]

rockbottom,

I usually leave all audio tracks on because I have several sources, I have my micropnone, the desktop audio, my discord passthrough and my browser passthrough (plus other stuff if I need it), I just edit these tracks out as I see fit in Vegas. Every audio track I've rendered with the NVENC encoder has come back fine, I don't think this is the issue. Thanks for your input.

 

[rockbottom]

You came here looking for help, testing/troubleshooting is part of determining what is causing the problem. Although this bug might not appear to apply to you, I asked you to record a single audio track since you ARE running your GPU/Monitors at 144fps.

[BUG] Multi track audio recording generates empty output file at high FPS · Issue #2855 · obsproject/obs-studio

Platform Operating system and version: Windows 10 1909 OBS Studio version: 25.0.8 Expected Behavior Recordings at high FPS should function normally. Current Behavior The OBS audio_thread is single ...

github.com

That said, having a Game Capture/Monitor Capture in the same scene can cause issues.
16:17:37.068: - scene 'Main Capture':
16:17:37.068: - source: 'Main Monitor' (monitor_capture)
16:17:37.068: - source: 'Supermodel 3' (game_capture)

You have some minor rendering lag, try running your monitors at 120fps to clear this up
16:39:20.902: Output 'adv_file_output': Total frames output: 57875
16:39:20.902: Output 'adv_file_output': Total drawn frames: 57854 (57918 attempted)
16:39:20.902: Output 'adv_file_output': Number of lagged frames due to rendering lag/stalls: 64 (0.1%)

Turn this off & enable Game Mode
16:17:34.889: Windows 10 Gaming Features:
16:17:34.889: Game DVR: On

 

[MarioGuy789]

rockbottom,

I apologize for getting back to you so long after you replied, and for writing your help off. I have since experimented with the NVENC encoder more and have since been able to run it without a hitch, even with multiple audio tracks running, though I'm not entirely sure what I did to make this possible. (I'll include a post of my latest diagnostic log).

To address the first point, I have moved all game and display captures into separate scenes so as to avoid any future conflictions.

To address the second point, I have not yet tried running my monitors at 120fps, as I have been able to get the encoder to work fine at 144, but I will do this in the future.

Lastly, I have Game Mode enabled already, but how do I go about turning the Windows Game DVR off? What is it listed under in the Windows settings? When I type "Game DVR" directly into the Windows search bar, the first thing that comes up is the Xbox Console Companion, which I don't believe is what is being referred to here, unless I'm mistaken.

Thanks for your time, and again, I apologize for being rude previously.

(Log: https://obsproject.com/logs/fwbYNCFnDrkv4gNg )

Addendum:

I am aware that the log says the game capture and monitor captures are still in the same scene, but they are not.

 

[Tomasz Góral]

Game DVR - look in Windows Settings 'Xbox Game Bar' and off.