Trouble recording high-fidelity games on top-end hardware

[hudzell]

I am by no means an encoding wiz, and I'm having trouble finding the right configuration spread for recording high-fidelity games.

My main issue is I get lots of dropped frames depending on how much action is on screen, for one game in particular this issue is extremely noticable, Monster Hunter: World

I have an RTX 2080Ti as my GPU and an i9-9900KF as my CPU, and I am using the NVENC encoder. I love recording my gameplay for clipping later, but a lot of the times the high action scenes I'd like to clip end up choppy, even when recording at just 720p30 with this hardware.

My current encoding settings are these

My goal and problem I'm trying to solve is not exactly image quality, but minimizing dropped frames while maintaining acceptable image quality.

With the right information I believe I could solve this on my own, cause I'm not entirely sure exactly how some of these encoding settings affect load, or even what they precisely do. I have a few questions:

• What's the exact difference between rescaling output in the Output tab vs. the Output (Scaled) Resolution option in the Video tab?
• How much does bitrate affect load?
• Which rate control setting would be optimal for me? How much do they matter?
• For the presets, what do all the different words mean exactly? Like Quality, Performance, Low-latency. Quality of what? Performance of what? The video or the encoder? And what does low-latency mean? Is it related to streaming?
• What exactly do the different profiles mean? high, main, and baseline are not very descriptive names.
• Are either Look-ahead and/or Psycho Visual Tuning good options for me?

 

[koala]

• your screenshot is from the streaming encoder settings, not from the recording settings
• rescaling output in the encoder settings takes place on the CPU, while rescaling in settings->output takes place on the GPU. GPU rescaling is far superior. Use rescaling in the encoder only if you absolutely must - usually only required if you want to stream and record with different resolutions at the same time. CPU rescaling leads to lost frames very easily
• if you don't need to record to multiple audio tracks, switch to simple output mode, then choose "indistinguishable quality" as recording quality. In advanced mode, use cqp as rate control and a cq value of about 15-20 (lower values are higher quality). Don't use CBR for recording. It's a rate control for streaming.
• the presets are an encoder configuration. Better quality means higher computing resource demand. You use the best setting your encoder is still able to process without issues. "Performance" means less computing demands of the encoder: "Quality" is the resulting image quality. With nvenc, this is the configuration of the hardware circuit that is nvenc. Higher resolutions and higher fps, such as 4k in 60 fps, might not be achieved with "Quality" preset, only with "Performance" preset. There are charts that tell how many fps nvenc is able to process depending on resolution and preset.
• profiles high, main and baseline define decoder requirements of the media player that is used for playing the video. Today, every player supports "high" profile, so use this. High profile enables all h.264 capabilities, so compression is best.
• Look-ahead and Psycho-visual tuning analyze motion in the video. The human eye isn't able to see as much detail with high motion than with low motion, so it's sensible to increase compression with high motion and decrease compression with low motion and get better perceived image quality with the same bitrate, or lower bitrate with the same perceived quality. Not so important with variable bitrate rate control like CQP. It's using up GPU resources (nvenc itself doesn't), so you might disable it if your GPU load becomes too high. With live streaming, it enhances quality.

 

[hudzell]

• your screenshot is from the streaming encoder settings, not from the recording settings
• rescaling output in the encoder settings takes place on the CPU, while rescaling in settings->output takes place on the GPU. GPU rescaling is far superior. Use rescaling in the encoder only if you absolutely must - usually only required if you want to stream and record with different resolutions at the same time. CPU rescaling leads to lost frames very easily
• if you don't need to record to multiple audio tracks, switch to simple output mode, then choose "indistinguishable quality" as recording quality. In advanced mode, use cqp as rate control and a cq value of about 15-20 (lower values are higher quality). Don't use CBR for recording. It's a rate control for streaming.
• the presets are an encoder configuration. Better quality means higher computing resource demand. You use the best setting your encoder is still able to process without issues. "Performance" means less computing demands of the encoder: "Quality" is the resulting image quality. With nvenc, this is the configuration of the hardware circuit that is nvenc. Higher resolutions and higher fps, such as 4k in 60 fps, might not be achieved with "Quality" preset, only with "Performance" preset. There are charts that tell how many fps nvenc is able to process depending on resolution and preset.
• profiles high, main and baseline define decoder requirements of the media player that is used for playing the video. Today, every player supports "high" profile, so use this. High profile enables all h.264 capabilities, so compression is best.
• Look-ahead and Psycho-visual tuning analyze motion in the video. The human eye isn't able to see as much detail with high motion than with low motion, so it's sensible to increase compression with high motion and decrease compression with low motion and get better perceived image quality with the same bitrate, or lower bitrate with the same perceived quality. Not so important with variable bitrate rate control like CQP. It's using up GPU resources (nvenc itself doesn't), so you might disable it if your GPU load becomes too high. With live streaming, it enhances quality.

Thank you! This is the information I needed.
For the first point, I know! My recording encoder is set to use the stream encoder, is there a reason not to do that?

I did a test recording with some new settings, even on 1080p60, and it appears to be perfectly smooth!

 

[koala]

You seem to want high quality recordings and you didn't mention you intend to stream, so you should not use the inferior encoding quality of streaming for recording. Set your recording encoder to nvenc (new) and configure CQP as rate control.

 

[hudzell]

You seem to want high quality recordings and you didn't mention you intend to stream, so you should not use the inferior encoding quality of streaming for recording. Set your recording encoder to nvenc (new) and configure CQP as rate control.

Is there really a difference? I was under the assumption that using the stream encoder for recording is basically just using the same settings for the recording encoder.

 

[koala]

It's the same encoder, but not the same encoding settings. Streaming needs constant bitrate, CBR rate control. High motion scenes need more space for encoding, because the difference between each frame is high. To achieve a constant bitrate, the encoder needs to dynamically increase the compression for high motion scenes, thus it gets worse quality. Blurry, washed out images during high motion. For low motion scenes, compression can be dynamically decreased, because difference between frames is low, so these look good.

For recording, bitrate doesn't need to be constant. So the encoder does all encoding with the same compression, so even high motion scenes are crystal clear. This is achieved by the CQP rate control.

"use stream encoder" doesn't inherit the encoder alone, it also inherits all settings including CBR rate control. It's basically a "encode for streaming, but instead of actually streaming write the resulting data to disk" setting.