seeking help with understanding key interval, multipass mode vs single, & b-frame setting

[witchcraft_streams]

Hi, I'm not experiencing any specific problem like lag or what have you. I'm just trying to set my OBS local recording settings to be the best quality they can be, even if it's a bit overkill or gets into diminishing returns territory. I can always gradually walk some settings back and compare quality tests that I record, but first I want to try and find or get very near to that theoretical quality ceiling with every setting. Even if the test result is an oversized laggy mess of a video file, at least then I know what happens and why kinda thing.

I will include a screenshot of my current OBS local recording settings, and some of my PC specs that I think might be relevant.

Mainly, I'd like to know what each of these three settings (keyframe int, multipass vs singlepass, b-frames) do; how or if each of them impact the visual quality of a recording (video game content); and how or if each setting would impact the file size, and resource intensiveness on GPU and/or RAM. Praying hands emoji.

PC specs:

Intel i9-12900k
NVIDIA 3080ti
32GB RAM G.Skill RipJaws V
and 2TB Samsung SSD (of which I have 300ish GB free, not worried about space though, I don't record super duper long gaming videos)

I record primarily in 1080p or 720p, and edit/render with non-pirated Vegas Pro 22.

I play a variety of games. Some can be quite intensive, like the new Indiana Jones game on high settings, or Helldivers 2. Some are not extremely intensive but have a lot of fast movement like Rocket League. Others are overall less intensive by comparison, like Persona.

 

[witchcraft_streams]

I forgot to mention in my title that I'm wondering about local recording specifically, not streaming. My bad.

 

[qhobbes]

Regarding Keyframes and B-Frames:
The two parameters deal with quality. They trade off space for quality. If you record with a quality-based rate control such as CQP or CRF, you have infinite space, so you can just optimize for quality if you want. B-frames are the ones with the highest compression (most detail removed), so the more B-frames you insert, the lower the quality. So to optimize B frames for quality, you should use 0 B-frames (none at all) with CQP.
With key frames, it's the same, only on a higher level and the other way round. They contain a whole frame and are an anchor for P-frames, which have a higher higher compression (less detail removed) than the keyframes (but lower than the B-frames). So if you want higher quality, use more keyframes, which can be achieved by using a smaller keyframe interval. It has the side effect that a video with more keyframes is better seekable. With lower keyframe interval, video size increases vastly.

With CBR rate control, the effect is reversed, since you limit the bitrate. To achieve the forced bitrate, the encoder removes as much detail as needed. If you don't use B-frames or use a lower keyframe interval, the bitrate is consumed completely by the bigger frames, so the general quality must be lowered, which is very visible. So don't do this (don't use CBR for recording).
Props to @koala for that.

NVENC Multi pass frame encoding​

When determining the QP to use for encoding a frame, it is beneficial if NVENC knows the overall complexity of the frame to distribute the available bit budget in the most optimal manner. In some situations, multi-pass encoding may also help catch larger motion between frames. For this purpose, NVENC supports the following types of multi-pass frame encoding modes:

• 1-pass per frame encoding (NV_ENC_MULTI_PASS_DISABLED)
• 2-passes per frame, with first pass in quarter resolution and second pass in full resolution (NV_ENC_TWO_PASS_QUARTER_RESOLUTION)
• 2-passes per frame, with both passes in full resolution (NV_ENC_TWO_PASS_FULL_RESOLUION).

In 1-pass rate control modes, NVENC estimates the required QP for the macroblock and immediately encodes the macroblock. In 2-pass rate control modes, NVENC estimates the complexity of the frame to be encoded and determines bit distribution across the frame in the first pass. In the second pass, NVENC encodes macroblocks in the frame using the distribution determined in the first pass. As a result, with 2-pass rate control modes, NVENC can distribute the bits more optimally within the frame and can reach closer to the target bitrate, especially for CBR encoding. Note, however, that everything else being the same, performance of 2-pass rate control mode is lower than that of 1-pass rate control mode. The client application should choose an appropriate multi-pass rate control mode after evaluating various modes, as each of the modes has its own advantages and disadvantages. NV_ENC_TWO_PASS_FULL_RESOLUION generates better statistics for the second pass, whereas NV_ENC_TWO_PASS_QUARTER_RESOLUTION results in larger motion vectors being caught and fed as hints to second pass.

 

[witchcraft_streams]

Regarding Keyframes and B-Frames:
The two parameters deal with quality. They trade off space for quality. If you record with a quality-based rate control such as CQP or CRF, you have infinite space, so you can just optimize for quality if you want. B-frames are the ones with the highest compression (most detail removed), so the more B-frames you insert, the lower the quality. So to optimize B frames for quality, you should use 0 B-frames (none at all) with CQP.
With key frames, it's the same, only on a higher level and the other way round. They contain a whole frame and are an anchor for P-frames, which have a higher higher compression (less detail removed) than the keyframes (but lower than the B-frames). So if you want higher quality, use more keyframes, which can be achieved by using a smaller keyframe interval. It has the side effect that a video with more keyframes is better seekable. With lower keyframe interval, video size increases vastly.

With CBR rate control, the effect is reversed, since you limit the bitrate. To achieve the forced bitrate, the encoder removes as much detail as needed. If you don't use B-frames or use a lower keyframe interval, the bitrate is consumed completely by the bigger frames, so the general quality must be lowered, which is very visible. So don't do this (don't use CBR for recording).
Props to @koala for that.

NVENC Multi pass frame encoding​

When determining the QP to use for encoding a frame, it is beneficial if NVENC knows the overall complexity of the frame to distribute the available bit budget in the most optimal manner. In some situations, multi-pass encoding may also help catch larger motion between frames. For this purpose, NVENC supports the following types of multi-pass frame encoding modes:

• 1-pass per frame encoding (NV_ENC_MULTI_PASS_DISABLED)
• 2-passes per frame, with first pass in quarter resolution and second pass in full resolution (NV_ENC_TWO_PASS_QUARTER_RESOLUTION)
• 2-passes per frame, with both passes in full resolution (NV_ENC_TWO_PASS_FULL_RESOLUION).

In 1-pass rate control modes, NVENC estimates the required QP for the macroblock and immediately encodes the macroblock. In 2-pass rate control modes, NVENC estimates the complexity of the frame to be encoded and determines bit distribution across the frame in the first pass. In the second pass, NVENC encodes macroblocks in the frame using the distribution determined in the first pass. As a result, with 2-pass rate control modes, NVENC can distribute the bits more optimally within the frame and can reach closer to the target bitrate, especially for CBR encoding. Note, however, that everything else being the same, performance of 2-pass rate control mode is lower than that of 1-pass rate control mode. The client application should choose an appropriate multi-pass rate control mode after evaluating various modes, as each of the modes has its own advantages and disadvantages. NV_ENC_TWO_PASS_FULL_RESOLUION generates better statistics for the second pass, whereas NV_ENC_TWO_PASS_QUARTER_RESOLUTION results in larger motion vectors being caught and fed as hints to second pass.

Thank you qhobbes (and also koala), you are a wizard, and I also love your profile pic.

My takeaways:

1. No b-frames is the "best" quality for local recording with CQP, but I assume it results in a significantly larger file size because there's way less compression.

2. A lower keyframe interval can help with "seeking" through the recording, which can be helpful in editing software. You will also have more anchors for p-frames (which are between keyframes and b-frames in terms of their compression and overall visual quality). I am wondering if zero, or auto, would be better than 1 keyframe per second. But I think that I will test 1 second interval and check the quality, file size, and compatibility in Vegas Pro 22 afterward. Lower intervals necessarily result in bigger file size because there are more "whole frames," literally double if going from 2 to 1.

3. You mention that multi-pass is "especially" optimal for CBR encoding. Since I am using CQP, then I would assume that the number of passes makes some degree of quality difference, just not as much as CBR. It also sounds like more individual testing of single or multi-pass might be necessary on the user's end compared to keyframes / b-frame settings which seem to have more straightforward impacts.

Correct me if I'm wrong, but it sounds like single pass "estimates the required QP" and then encodes without any further checks (so to speak), which is less resource intensive but has a higher probability of getting it wrong—for lack of a better word—because it lacks a second pass, potentially either allocating too much or too little. Two pass quarter is a little more optimal, using a first pass to gather information (which is good enough for catching "larger motion between frames") that can then be used for a more improved second pass, but more resource intensive for the simple fact that it is running these second passes. And two pass full res would overall get the closest to "optimal" bitrate allocation because it's using two full resolution passes, but at the cost of much more resource intensiveness / performance?

Hmm... I might take the middle road here and go for multi-pass quarter res. I'll test that in a few different games, and if it all looks good than I'll leave it at that.

Thanks again! :) Also appreciate the explanations of each setting, I found it easy to understand what each one generally does, which made it easier to understand what happens when you make changes / adjustments to them.