Question / Help ELI5: How does NVENC work?

[Tawm]

I'm interested in how NVENC and x264 do their encoding, and why people say NVENC is so horrible for bitrates under 30,000 (depending on resolution and framerate, I suppose).

My assumption is that NVENC and x264 do the same job but at different speeds and create different file sizes.
Based on how a CPU and GPU are used for processing, I'm thinking that x264 works harder to create a smaller file while NVENC works less hard and packs the same quality in a much larger file.

 

[Boildown]

NVEnc is close to a fixed function hardware device. It does one thing well. x264 is software and more flexible. NVEnc's one thing is high bitrate encoding. x264 can do anything given CPU time. If you starve NVEnc of the bitrate it needs, it looks horrible. x264 can still look decent at low bitrates. At high bitrates they're about equivalent, but because x264 uses the CPU instead of the GPU, if you want to keep your CPU free for other tasks you might prefer to use NVEnc over x264 for high bitrate encoding.

As an aside, I found NVEnc recordings to look acceptable at 20Mb/s at 1920x1080x60fps. But I could tell a quality difference up to 30Mb/s. Normally I record at 30Mb/s unless I feel the need to save hard drive space. I never stream with NVEnc because it requires too much bitrate to look good.

 

[Tawm]

@Boildown thank you for that.
This is a bit different, but if I wanted to both record and stream at the same time - both using x264 - does the CPU have to work twice as hard?
What about NVEnc recording and x264 stream?

I doubt I'd bother with it, but I'm interested nonetheless.

 

[Xaymar]

I have done tests with NVENC and x264 quality, see my work here. I gave the same lossless 1080p60 input to either encoder (automated, through ffmpeg) and then had ffmpeg generate SSIM measurements for the results. The result is that NVENC is almost as good as x264 veryfast, only being 0.3204%pt worse at 6000kbit (twitch maximum), 0.522%pt worse at 3500kbit (old twitch recommended) or 0.6504%pt worse at 3500kbit. Using NVENC on Pascal GPU is decent enough for streaming nowadays - maybe with the next generation it will beat x264 veryfast.

Now to the other question. Unless you set the recording tab to "use stream encoder" your CPU will do double the work. You can mix both x264 and NVENC.

 

[DeMoN]

As an aside, I found NVEnc recordings to look acceptable at 20Mb/s at 1920x1080x60fps. But I could tell a quality difference up to 30Mb/s. Normally I record at 30Mb/s unless I feel the need to save hard drive space.

I wonder what games you are capturing that you dont see any loss at 30mbit and NVEnc.
At least with a vegetation heavy game, or F1 2016 or Dirt 4 even the blindest man on earth should see the strong quality loss xD

 

[Boildown]

I wonder what games you are capturing that you dont see any loss at 30mbit and NVEnc.
At least with a vegetation heavy game, or F1 2016 or Dirt 4 even the blindest man on earth should see the strong quality loss xD

That's not what I said. I said I could still tell a distinct quality difference (improvement) by encoding at 30Mb/s instead of 20Mb/s. I did not mean to imply that I thought 30Mb/s was transparent to the source. It isn't. It was Planetside 2, by the way. I will say that for the games I've recorded so far 30Mb/s is close enough to transparent that I don't think its worth it to increase the bitrate further without a specific use for the recordings in mind, so I keep it at 30Mb/s.

 

[Boildown]

I have done tests with NVENC and x264 quality, see my work here. I gave the same lossless 1080p60 input to either encoder (automated, through ffmpeg) and then had ffmpeg generate SSIM measurements for the results. The result is that NVENC is almost as good as x264 veryfast, only being 0.3204%pt worse at 6000kbit (twitch maximum), 0.522%pt worse at 3500kbit (old twitch recommended) or 0.6504%pt worse at 3500kbit. Using NVENC on Pascal GPU is decent enough for streaming nowadays - maybe with the next generation it will beat x264 veryfast.

That is some fascinating results. Did you happen to test any Maxwell, and if so, how much worse are they than Pascal? Did you post your results at the lower bitrates? I only see the 6000kb/s tests.

Oh yeah, the "Two Pass" option in NVEnc-OBS... what does it actually do? It looks like you have it disabled for constant bitrate but I don't think that's required in the OBS GUI. (Two-pass doesn't actually make sense in live video encoding but the fact that we can enable it and live stream means it either does nothing or it does something other than what it implies.)

Thanks for doing all that work!

 

[Xaymar]

That is some fascinating results. Did you happen to test any Maxwell, and if so, how much worse are they than Pascal?

Some Maxwell cards are supposed to be better than Pascal, but I'd expect that the quality is about the same for both.

Did you post your results at the lower bitrates? I only see the 6000kb/s tests.

The default filter shows 6000kbit only, as that is the twitch maximum (and what I would recommend for 1080p60). You can change the filter by creating a copy of the Google Drive sheet. :)

Oh yeah, the "Two Pass" option in NVEnc-OBS... what does it actually do? It looks like you have it disabled for constant bitrate but I don't think that's required in the OBS GUI. (Two-pass doesn't actually make sense in live video encoding but the fact that we can enable it and live stream means it either does nothing or it does something other than what it implies.)

My test results showed that enabling two pass doesn't actually do anything with CBR, only had an effect when using VBR. I think it'll be the same for OBS too, since it uses it through ffmpeg. As far as I am aware, HW encoding has no true multi pass encoding, it's more of a look-ahead type encoding.

 

[DeMoN]

2-pass rate control modes help the encoder to gather statistics of the frame to be encoded before actually encoding it in the second pass, thereby resulting in optimal bit-utilization within the frame and consequently, higher encoding quality.

From here: http://developer.download.nvidia.com/compute/nvenc/v4.0/NVENC_AppNote.pdf

2pass and bframes may cause some framedrop and/or more ingamefps loss, I would rather use it at an file encode instead of a capture.

 

[Tawm]

I gave the same lossless 1080p60 input to either encoder (automated, through ffmpeg) and then had ffmpeg generate SSIM measurements for the results.

I keep thinking about this quote.
I keep going back and forth between NVEnc and x264 settings and looking at the resulting video captures to see which is better. I just did some basic research on SSIM and now realize that maybe I can know that one video capture is actually better than the other. However, I don't know how I'd capture a Lossless (perfect) video and then a regular capture.
I don't think my machine can record Lossless without having a laggy resulting video. Also, I don't think it's meant to be compared to a video game, more like a source video.

I've got a brand new GPU (1080Ti) and a 2gen-old CPU (i7-5820k), so the resulting NVEnc and x264 encoded videos about the same while pretty much maxing out the related Processing Unit.

Based on my setup, can you think of anything I'd be able to do to see the numerical results of a capture so that I can compare my recordings from different settings?

 

[Boildown]

Based on my setup, can you think of anything I'd be able to do to see the numerical results of a capture so that I can compare my recordings from different settings?

When doing x264 encoding, I just trust that slower presets are higher quality at a given bitrate than faster ones. So then I look at my duplicated/lagged/whatevered frames in the OBS log and see if they're unacceptably high. And if they're near or at zero, then I drop the preset a notch slower and see how the frame statistics react.

For NVEnc encoding, I don't really care, I just give it enough bitrate to look really close to the original and still check the frame statistics in the OBS log to make sure something isn't messing up. Of course, I don't stream with NVEnc, so I don't have to concern myself too much with it.

 

[Tawm]

When doing x264 encoding, I just trust that slower presets are higher quality at a given bitrate than faster ones.
For NVEnc encoding, I don't really care, I just give it enough bitrate to look really close to the original...

I've normally been using CRF or whatever the equivalent is for NVEnc instead of an actual bitrate. I feel like CRF will give a better overall quality instead of setting a specific bitrate.

So now that I think of it. I suppose it really doesn't matter what x264 preset I'd use as long as I'm using CRF, since the level of quality will be at the same "number", regardless of if I'm using superfast or ultrafast. Is this accurate?

And I suppose I'll have to look at the log after a capture session to find these "frame statistics".

 

[Boildown]

Constant quality or rate factor encoding is fine for recordings to disk, but for streaming you need to run in Constant Bit Rate (CBR) mode. You can also use CBR for recording to disk but it could be / probably is less efficient in a few ways, but it isn't bad. It is bad to do anything but CBR for streaming, due to how IP network traffic works.

Unless you're super-technical about it, you're right about CRF and the various presets. x264 will (mostly) just adjust the bitrate on the fly to maintain a certain quality level. Ultrafast is one of those technical points though, it turns off some x264 features that can't entirely be compensated for with more bitrate or better quality CRF settings. Try to stick to SuperFast if at all possible.

For NVEnc, I'm a lot less familiar with its variable bitrate options, but regardless I wouldn't use them for internet streaming for the same reasons as above.

 

[Tawm]

Ultrafast is one of those technical points though, it turns off some x264 features that can't entirely be compensated for with more bitrate or better quality CRF settings. Try to stick to SuperFast if at all possible.

I've read in other threads that Superfast is considered the "sweet spot" but I never got a great explanation for why. Could you explain what features are turned off or provide a link so that I could read up on it?

I don't think my CPU can handle running Superfast when capturing a CPU-heavy game (Overwatch is very CPU dependent) at 1440p60, so I will likely not start using it, but I'd like to see what x264 does at higher speeds and maybe I can find similar info for NVEnc.

For the past few days, NVEnc has been working very well for me on games like Overwatch and PUBG. It results in some great looking captures while pegging my GPU around 90-100%, which I've read GPUs are designed to handle, so I'm not too worried about damage. When using high x264 settings, however, my CPU is also pegged around 90-100%, but that has some significant negative impacts on my overall computer performance for obvious reasons.

 

[Boildown]

This is no longer ELI5 compatible... http://dev.beandog.org/x264_preset_reference.html

CABAC is the big one from what I've read. UltraFast is the only preset to turn it off and fallback to the crappier method.

You could set UltraFast preset and then use x264 custom commands to find some intermediate config (that turns on CABAC at the very least) that your CPU can handle I suppose. That would be nearly as far from ELI5 as you can get in OBS though. Lots of experimentation would be needed. I've done this between other presets, between Fast and Medium, and some intermediate states look like complete garbage because some features should be enabled as a group and there's no documentation on what those might be.

Why even bother though? According to Xaymar's tests NVEnc on HQ preset is better than SuperFast preset anyways. So just stream with that.

 

[Tawm]

This is no longer ELI5 compatible...

Thanks for that link. And sorry for the evolving thread; I find it easiest to keep a train of thought going instead of answering a thread and starting a new one. But perhaps this is splitting a bit too far from the original question.

Thanks for the help with my questions.
I'll likely start looking into NVEnc settings and tweaks.

 

[DeMoN]

Unless you're super-technical about it, you're right about CRF and the various presets. x264 will (mostly) just adjust the bitrate on the fly to maintain a certain quality level. Ultrafast is one of those technical points though, it turns off some x264 features that can't entirely be compensated for with more bitrate or better quality CRF settings. Try to stick to SuperFast if at all possible.

I'd like to add that a faster preset also means that the CRF loses a bit of accuracy, because the encoder will find less details with a less intense search. You'll often notice that a CRF encode of very fast has a smaller filesize than a preset slow file with same CRF. That sounds illogical, but it happens quite often and is the result of overlooked details.

 

[Tawm]

...a faster preset... will find less details with a less intense search... a CRF encode of very fast has a smaller filesize than a preset slow file with same CRF. ...it happens quite often and is the result of overlooked details.

That would tell me that a slower preset should look better with more details, even at the same CRF. Can you explain why? Wouldn't a faster preset result in a larger filesize but keep all details (based on the CRF) while a slower preset will result in a smaller filesize but still all details (CRF)?

 

[DeMoN]

Yes.
But reread what I wrote. I've explained what you have to consider additional to this.
But you could decrease the CRF Value by 1 to compensate against the not found detail due to the rough search of the faster presets.
Generally it sounds worse than it actually is, but at lower bitrates the story will be more relevant.

 

[Boildown]

As presets improve they only try to increase the amount of quality for the given amount of bit usage. They can do this by increasing quality using the same amount of bits, increasing quality a lot even though the bits have increased a little, they can decrease the quality a little while decreasing the bits a lot. Or probably combinations of the above that are too numerous to list. There's no actual guarantee that at a given CRF or bitrate that a better preset will have better quality and/or smaller filesizes. It'll just be better (if no one made any mistakes) as a ratio of the two compared to the preset faster than it.

That's the ELI-advanced-hobbyist explanation as I understand it anyways.