Hello. I've been using OBS Studio for my local recordings and occasional Twitch streaming for about a year now. In the course of tens of hours of learning about encoding, nvenc's encoders, and the deep rabbit hole known as ffmpeg, I've uncovered an error in OBS Studio regarding hevc using b-frames as refrerences, in ffmpeg custom output.
The problem: When recording using NVENC HEVC (I've tried the mp4 and mkv containers), neither custom ffmpeg output, nor Xaymar's StudioFX ffmpeg output, work when HEVC b_ref_mode is set to "middle". Output does work correctly when set to "disabled"
The reason: I believe the issue is related issue found here: https://trac.ffmpeg.org/ticket/7303 My post in the link goes to example video files and ffmpeg logs with the error displayed. It's related to the PTS and DTS not matching when using HEVC b-frames as references. The ffmpeg developers consider the issue fixed, as the workaround they implemented produces acceptable video files with ffmpeg cli.
Unfortunately, the OBS Studio ffmpeg multiplexer does not know how to handle the error, and does not encode properly. Attached are logs containing successful H264 recordings, and both successful and unsuccessful H265 recordings.
I'm using the latest official NVIDIA DCH driver 445.87, but this problem has occurred for over a year across many OBS Studio and Nvidia driver versions.
Testing was done on a fresh install of OBS Studio 25.0.4.
The problem: When recording using NVENC HEVC (I've tried the mp4 and mkv containers), neither custom ffmpeg output, nor Xaymar's StudioFX ffmpeg output, work when HEVC b_ref_mode is set to "middle". Output does work correctly when set to "disabled"
The reason: I believe the issue is related issue found here: https://trac.ffmpeg.org/ticket/7303 My post in the link goes to example video files and ffmpeg logs with the error displayed. It's related to the PTS and DTS not matching when using HEVC b-frames as references. The ffmpeg developers consider the issue fixed, as the workaround they implemented produces acceptable video files with ffmpeg cli.
Unfortunately, the OBS Studio ffmpeg multiplexer does not know how to handle the error, and does not encode properly. Attached are logs containing successful H264 recordings, and both successful and unsuccessful H265 recordings.
I'm using the latest official NVIDIA DCH driver 445.87, but this problem has occurred for over a year across many OBS Studio and Nvidia driver versions.
Testing was done on a fresh install of OBS Studio 25.0.4.
