Bug Report OBS Studio Crashes after versions 0.16.3-

[Maginhart]

I'm streaming and doing local recording at the same time using x264 for stream and NVENC encoder for local recording

Problems began after i've installed 0.16.3 version, when i'm trying to launch local recording with stream at the same time obs crashes.
On 0.16.2 version everything works flawlessly

Here's part of my log file after crash

10:25:37.146: [NVENC encoder: 'recording_h264'] settings:
10:25:37.146: rate_control: VBR
10:25:37.146: bitrate: 50000
10:25:37.146: cqp: 23
10:25:37.146: keyint: 250
10:25:37.146: preset: bd
10:25:37.146: profile: main
10:25:37.146: level: auto
10:25:37.146: width: 1920
10:25:37.146: height: 1080
10:25:37.146: 2-pass: true
10:25:37.146: b-frames: 2
10:25:37.146: GPU: 0
10:25:37.146:
10:25:37.147: warning: This encoder is deprecated, use 'h264_nvenc' instead
10:25:37.220: warning: The variable bitrate rate-control requires the 'qmin' and/or 'qmax' option set.
10:25:37.277: error: CreateBitstreamBuffer failed: out of memory (10)

 

[Maginhart]

Full log file

 

[RytoEX]

As the log notes:

10:25:33.659: [NVENC encoder: 'recording_h264'] settings:
10:25:33.659: rate_control: VBR
10:25:33.659: bitrate: 50000
10:25:33.659: cqp: 23
10:25:33.659: keyint: 250
10:25:33.659: preset: bd
10:25:33.659: profile: main
10:25:33.659: level: auto
10:25:33.659: width: 1920
10:25:33.659: height: 1080
10:25:33.659: 2-pass: true
10:25:33.659: b-frames: 2
10:25:33.659: GPU: 0
10:25:33.659:
10:25:33.659: warning: This encoder is deprecated, use 'h264_nvenc' instead
10:25:33.812: warning: The variable bitrate rate-control requires the 'qmin' and/or 'qmax' option set.
10:25:33.886: error: CreateBitstreamBuffer failed: out of memory (10)
10:25:33.941: [NVENC encoder: 'recording_h264'] Failed to open NVENC codec: Cannot allocate memory

OBS can't allocate enough memory for the NVENC encoder. You could try switching to the 64-bit OBS Studio to allow it to access more memory. You might also consider splitting your current Scene Collection into several smaller Scene Collections so that OBS uses less memory at once (sources within a Scene or Scene Collection are global in OBS Studio).

You might also consider adjusting your NVENC recording settings to use CQP instead of VBR. See this guide for recommendations on recording settings.