NVENC vs. x264 for local recording - Some thoughts

[Guilherme Hoffmann]

TL;DR: in my experience x264 is much better than Nvenc at recording high resolution, high frame rate, heavy GPU load games

So, I used to have a 1080p monitor and I recorded 1080p60 with Nvenc.
I now switched to a 1440p monitor, and after recording some 1440p60 gameplay and putting on Youtube, it's hard to go back. I spend roughly twice the time to render a video in this resolution, but seeing that quality in a video later really pays off.

Thing is, when I started testing OBS recording in this resolution, I had a really hard time to figure out the best way.
Nvenc was dropping a lot of frames - even though OBS shows constant 60.00 fps at the bottom, the video file skips several frames, going as low as 39 fps in some moments.

After some testing I found out that in games where the GPU gets a 95-99% load, OBS struggles to pick all frames with Nvenc.
I tried limiting in game frame rate to something like 80, so that I could keep it loading only 80-85% in the GPU. This apparently did the job and reduced a lot of the frame dropping in video file. People say that Nvenc uses an exclusive sector of the GPU, but then why is this happening?

Anyway, with the x264 Ultrafast preset I was able to record an almost perfect constant 60 fps video without having to lower framerate in game. Thus, x264 is much better than Nvenc.

This was all tested in Battlefield 1 and Overwatch. I'm pretty sure it's gonna vary depending on the game though.

Any thoughts on this? Is it normal for Nvenc to fail at heavy GPU load?

 

[Lapppy]

In my experience, a GPU under heavy load will cause frame skips regardless of the chosen encoder since the problem is when the frames are being captured (game capture) rather than when they are encoded.
The only difference between NVENC and x.264 in my case is encoding performance, and NVENC easily wins in this department on my setup.

 

[RytoEX]

After some testing I found out that in games where the GPU gets a 95-99% load, OBS struggles to pick all frames with Nvenc.
I tried limiting in game frame rate to something like 80, so that I could keep it loading only 80-85% in the GPU. This apparently did the job and reduced a lot of the frame dropping in video file. People say that Nvenc uses an exclusive sector of the GPU, but then why is this happening?

First, x264 produces better quality video per bitrate/filesize than hardware encoders. That said, the hardware encoders can offer advantages in other areas (resource usage) that make them a better choice in some scenarios.

Now, why does GPU load affect OBS Studio? Because OBS uses the GPU to composite/render the scenes before they are encoded. The workflow goes something like this: gather all sources -> composite on GPU -> show preview -> encode -> output. Since some GPU usage and bandwidth are necessary no matter what encoder you choose, if the GPU is already burdened rendering the game, there may not be enough resources left for OBS Studio to do its work.

NVENC, as far as I know, does use an exclusive chip for encoding. However, the data still has to be sent to that chip, meaning bandwidth must be available to send data for encoding, and then receive encoded data.

In summary, if you're maxing out any one component in the encoding equation, expect sub-optimal results.

Anyone with more intimate knowledge of NVENC or encoding in general is welcome to provide more illumination here or correct any inaccuracies in my post.

 

[Suslik V]

Possibly, by capturing is named copy frame function, so more frames in game your gpu renders - more bandwidth used. Limit frame rate of the game.