Comparison of x264, NVENC, Quicksync, VCE

T

Toasty27

New Member
Interesting results. Just to clarify, is this with the NVENC engine on the 10-series cards, or the 20-series? Also, will you be posting results for the 2500kbps and 3500kbps encodes as well?
 
Xaymar

Xaymar

Active Member
Toasty27 said:
Interesting results. Just to clarify, is this with the NVENC engine on the 10-series cards, or the 20-series? Also, will you be posting results for the 2500kbps and 3500kbps encodes as well?
Click to expand...

These results are with the Turing series (20xx). My old tests with the Pascal series (10xx) are still available here, however I did not follow proper upscaling requirements for comparison in those and treated the results as "same resolution input, same resolution encode, same resolution output", which means that the scores aren't correct. I have a Laptop with a 1050 (not a M1050, an actual 1050) that I could use for retesting 10xx, but I don't think it'll be useful - back then it barely managed to beat veryfast, and it's likely to be the same now.

The remaining results are all in a database, I just have to make the API and website work.
 
Xaymar

Xaymar

Active Member
As asked, here are the results for 2.5mbit and 3.5mbit. One thing to note is that at those bitrates, many options have no effect as the compression hits a hard limit with this scene, and would need more bitrate to go with the frames. For example NVEncs Two-Pass either completely fails or reduces quality, which is the exact opposite of what we need. The list will only show VMAF scores, as I don't want to bother with PSNR and SSIM right now.

3.5mbit 1080p
  1. ue4-cabininthewoods/60/1920x1080/Partial/3500/nve264/$bf$4$rc$cbr_hq$2p$0$brm$0$saq$1$taq$1$rcla$32 55.832
  2. ue4-cabininthewoods/60/1920x1080/Partial/3500/nve264/$bf$4$rc$cbr_hq$2p$1$brm$0$saq$1$taq$1$rcla$32 55.811
  3. ue4-cabininthewoods/60/1920x1080/Partial/3500/nve264/$bf$4$rc$cbr_hq$2p$1$brm$0$saq$1$taq$1$rcla$0 55.52
  4. ue4-cabininthewoods/60/1920x1080/Partial/3500/nve264/$bf$4$rc$cbr_hq$2p$0$brm$0$saq$1$taq$1$rcla$0 55.52
  5. ue4-cabininthewoods/60/1920x1080/Partial/3500/nve264/$bf$4$rc$cbr_hq$2p$1$brm$2$saq$1$taq$1$rcla$4 55.059
  6. ue4-cabininthewoods/60/1920x1080/Partial/3500/nve264/$bf$4$rc$cbr_hq$2p$1$brm$2$saq$1$taq$1$rcla$32 55.059
  7. ue4-cabininthewoods/60/1920x1080/Partial/3500/nve264/$bf$4$rc$cbr_hq$2p$1$brm$2$saq$1$taq$1$rcla$16 55.059
  8. ue4-cabininthewoods/60/1920x1080/Partial/3500/nve264/$bf$4$rc$cbr_hq$2p$0$brm$2$saq$1$taq$1$rcla$32 55.059
  9. ue4-cabininthewoods/60/1920x1080/Partial/3500/nve264/$bf$4$rc$cbr_hq$2p$1$brm$2$saq$1$taq$1$rcla$1 55.052
  10. ue4-cabininthewoods/60/1920x1080/Partial/3500/nve264/$bf$4$rc$cbr_hq$2p$1$brm$2$saq$1$taq$1$rcla$8 55.021
  11. ue4-cabininthewoods/60/1920x1080/Partial/3500/nve264/$bf$4$rc$cbr_hq$2p$1$brm$2$saq$1$taq$1$rcla$2 55.02
  12. ue4-cabininthewoods/60/1920x1080/Partial/3500/nve264/$bf$4$rc$cbr_hq$2p$1$brm$2$saq$1$taq$1$rcla$0 54.983
  13. ue4-cabininthewoods/60/1920x1080/Partial/3500/nve264/$bf$4$rc$cbr_hq$2p$0$brm$2$saq$1$taq$1$rcla$0 54.983
  14. ue4-cabininthewoods/60/1920x1080/Partial/3500/x264/$bf$-1$preset$slow$nr$0$psy$1 52.172
2.5mbit 1080p
  1. ue4-cabininthewoods/60/1920x1080/Partial/2500/nve264/$bf$4$rc$cbr_hq$2p$0$brm$0$saq$1$taq$1$rcla$32 49.593
  2. ue4-cabininthewoods/60/1920x1080/Partial/2500/nve264/$bf$4$rc$cbr_hq$2p$1$brm$0$saq$1$taq$1$rcla$32 49.593
  3. ue4-cabininthewoods/60/1920x1080/Partial/2500/nve264/$bf$4$rc$cbr_hq$2p$0$brm$0$saq$1$taq$1$rcla$0 49.168
  4. ue4-cabininthewoods/60/1920x1080/Partial/2500/nve264/$bf$4$rc$cbr_hq$2p$1$brm$0$saq$1$taq$1$rcla$0 49.168
  5. ue4-cabininthewoods/60/1920x1080/Partial/2500/nve264/$bf$4$rc$cbr_hq$2p$0$brm$2$saq$1$taq$1$rcla$32 48.24
  6. ue4-cabininthewoods/60/1920x1080/Partial/2500/nve264/$bf$4$rc$cbr_hq$2p$1$brm$2$saq$1$taq$1$rcla$32 48.24
  7. ue4-cabininthewoods/60/1920x1080/Partial/2500/nve264/$bf$4$rc$cbr_hq$2p$0$brm$2$saq$1$taq$1$rcla$0 48.115
  8. ue4-cabininthewoods/60/1920x1080/Partial/2500/nve264/$bf$4$rc$cbr_hq$2p$1$brm$2$saq$1$taq$1$rcla$0 48.115
  9. ue4-cabininthewoods/60/1920x1080/Partial/2500/x264/$bf$-1$preset$slow$nr$0$psy$1 44.965
  10. ue4-cabininthewoods/60/1920x1080/Partial/2500/x264/$bf$-1$preset$slow$nr$16$psy$1 44.825
3.5mbit 720p
  1. ue4-cabininthewoods/60/1280x720/Partial/3500/nve264/$bf$4$rc$cbr_hq$2p$1$brm$0$saq$1$taq$1$rcla$32 56.661
  2. ue4-cabininthewoods/60/1280x720/Partial/3500/nve264/$bf$4$rc$cbr_hq$2p$0$brm$0$saq$1$taq$1$rcla$32 56.661
  3. ue4-cabininthewoods/60/1280x720/Partial/3500/nve264/$bf$4$rc$cbr_hq$2p$1$brm$0$saq$1$taq$1$rcla$0 56.546
  4. ue4-cabininthewoods/60/1280x720/Partial/3500/nve264/$bf$4$rc$cbr_hq$2p$0$brm$0$saq$1$taq$1$rcla$0 56.546
  5. ue4-cabininthewoods/60/1280x720/Partial/3500/nve264/$bf$4$rc$cbr_hq$2p$0$brm$2$saq$1$taq$1$rcla$32 56.464
  6. ue4-cabininthewoods/60/1280x720/Partial/3500/nve264/$bf$4$rc$cbr_hq$2p$1$brm$2$saq$1$taq$1$rcla$32 56.464
  7. ue4-cabininthewoods/60/1280x720/Partial/3500/nve264/$bf$4$rc$cbr_hq$2p$0$brm$2$saq$1$taq$1$rcla$0 56.365
  8. ue4-cabininthewoods/60/1280x720/Partial/3500/nve264/$bf$4$rc$cbr_hq$2p$1$brm$2$saq$1$taq$1$rcla$0 56.365
  9. ue4-cabininthewoods/60/1280x720/Partial/3500/x264/$bf$-1$preset$slow$nr$0$psy$1 55.908
  10. ue4-cabininthewoods/60/1280x720/Partial/3500/x264/$bf$-1$preset$slow$nr$16$psy$1 55.826
2.5mbit 720p
  1. ue4-cabininthewoods/60/1920x1080/Partial/2500/nve264/$bf$4$rc$cbr_hq$2p$0$brm$0$saq$1$taq$1$rcla$32 52.026
  2. ue4-cabininthewoods/60/1920x1080/Partial/2500/nve264/$bf$4$rc$cbr_hq$2p$1$brm$0$saq$1$taq$1$rcla$32 52.026
  3. ue4-cabininthewoods/60/1920x1080/Partial/2500/nve264/$bf$4$rc$cbr_hq$2p$0$brm$0$saq$1$taq$1$rcla$0 51.892
  4. ue4-cabininthewoods/60/1920x1080/Partial/2500/nve264/$bf$4$rc$cbr_hq$2p$1$brm$0$saq$1$taq$1$rcla$0 51.892
  5. ue4-cabininthewoods/60/1920x1080/Partial/2500/nve264/$bf$4$rc$cbr_hq$2p$0$brm$2$saq$1$taq$1$rcla$32 51.554
  6. ue4-cabininthewoods/60/1920x1080/Partial/2500/nve264/$bf$4$rc$cbr_hq$2p$1$brm$2$saq$1$taq$1$rcla$32 51.554
  7. ue4-cabininthewoods/60/1920x1080/Partial/2500/nve264/$bf$4$rc$cbr_hq$2p$0$brm$2$saq$1$taq$1$rcla$0 51.445
  8. ue4-cabininthewoods/60/1920x1080/Partial/2500/nve264/$bf$4$rc$cbr_hq$2p$1$brm$2$saq$1$taq$1$rcla$0 51.445
  9. ue4-cabininthewoods/60/1920x1080/Partial/2500/x264/$bf$-1$preset$slow$nr$0$psy$1 50.672
  10. ue4-cabininthewoods/60/1920x1080/Partial/2500/x264/$bf$-1$preset$slow$nr$16$psy$1 50.587
So with all of this data, something becomes clearer:
  • NVEnc ends up better in VMAF than x264 if the bitrate is less than the amount needed for x264 to create a clean compression. That means that NVEnc will do more drastic compression than x264 is willing to do in any profile.
  • However if the bitrate exceeds the necessary bitrate, x264 will jump ahead and dominate the field in VMAF. This can be seen in the original post, where x264 slow was at the top of 720p 6.0mbit.
  • NVEnc Two-Pass has zero impact on quality if the bitrate is too restrictive, and only takes away GPU time at that point. It might even cause lower quality, so watch out for that one.
I can't confirm or deny if setting B-Frames to 2 improves quality further. I know that B-Frame 1 is below x264, and B-Frame 4 is above x264 in bitrate restricted situations, so it really just comes down to how much GPU time you have available, and how much bitrate.
 
Xaymar

Xaymar

Active Member
I've finally gotten around to writing a web page for the encoding data, so you can now filter by file, resolution, framerate and bitrate! Also started comparing quality on additional files, and might include AMD Vega in a future update to the data: https://xaymar.com/ves/
 
koenigseggrs25

koenigseggrs25

New Member
Ok, first of all, this is a lot to ask, but:

I'm running a Nvidia GTX 960M, with 8gb of RAM and a Intel i5-6300HQ, should I use NVENC H.264 or QSV H.264. Yes, my laptop is old, and I play CPU demanding games (eg. Forza Horizon 4), and was wondering for recording which one I should use, and which preset. (x264 is not an option, my computer will probably kill itself). I record at 30fps, downscaled to 720p. As well, what should my bitrate be?
 
You must log in or register to reply here.
Top