Comparison of x264, NVENC, Quicksync, VCE

[koala]

The question we all want to know is:

What encoder and which encoder setting should I use for streaming/for recording?

and:

If I use a hardware encoder, does the quality suffer?

There are so many configuration options and there is so many different hardware available, so this is always very difficult to answer.

So I made a autohotkey script and created recordings of literally every combination of every encoder setting. Then I compared the quality of the resulting videos to the original lossless recording. The first level of comparison was made with ffmpeg (method see down below).

This is the result for OBS simple settings for streaming. I did the tests on 2 machines:

- i5-3570k with GTX 670 (QSV_IvyBridge with NVENC_Kepler)
- i7-6700k with GTX 1070 (QSV_Skylake with NVENC_Pascal)
- [no AMD-VCE, since I don't own one.] see post #25 for a comparison that includes VCE

Bitrates tested: 2000, 2500, 3500
Resolutions tested: 1280x720 (30 and 60 fps), 1920x1080 (30 fps)

The report only includes the one preset that gives the best quality for each encoder. This was done to not clutter the report with settings one will not use anyway. However, I tested all presets (for example for NVENC such as: "Low-Latency", "Low-Latency High Performance").

Excel-Sheet

From this computer-generated rating with mainly the mse as criteria, you may come to the conclusion that NVENC is on par with x264 preset=veryfast (the default in OBS), or even a bit better, but unfortunately it isn't. At least for high motion scenes.

Download the bitrate 2500 x264 and nvenc video and compare them with your eyes, and you will see more details in the second half of the video with x264 than with NVENC. I was mislead by the mse numbers myself, but I cannot deny the x264 video definitely shows a bit more more detail in high motion parts. Low motion looks the same with both.

Still correct is the ordering of NVENC and Quicksync: NVENC provides better visual quality than Quicksync. Quicksync on Skylake is better than on IvyBridge, but NVENC is still a bit better than both.

So I recommend the encoders in this order:

1. if your computer is able to encode x264 preset=veryfast or better while running your game, use this. This will interfere with your game and your stream, if your CPU is not powerful enough. Only desktop i5 and i7 processors are probably able to encode better than veryfast while a CPU-demanding game is run in parallel.
   
   
2. if you have NVENC available, use NVENC with preset=highquality. This will not interfere with your game at all.
   
   
3. if you have Quicksync available, use Quicksync with preset=balanced. This will not interfere with your game at all.
   
   
4. don't use x264 preset=superfast or ultrafast for streaming, as they produce way worse output than even Quicksync.

rating:

x264 faster = 80/100
x264 veryfast = 75/100
nvenc highquality = 70/100
quicksync balanced = 60/100
x264 superfast = 40/100
x264 ultrafast = 30/100​

***​

You can download the encoded videos here and do a visual comparison yourself:

http://www.wombaz.de/uploads/2016/11/obs-codec-test/

A convenient table for visual comparison between codecs is in this post:
https://obsproject.com/forum/posts/252221

If you do extensive visual comparison, please don't watch the videos though the browser but do a "Save as" and download them to your disk. Then watch them with your media player locally.
In the directories included are "analyis2.txt" files, which contain the machine-generated ratings from every single test.

***​

Spoiler: The method

The method:

• I created a 2 minute long reference video of my favorite game. 2560x1440p60 lossless recording with OBS (simple output mode->"Lossless Quality, Tremendously Large File Size"). The video includes 2 low motion scenes and 2 high motion scenes, each about the same length.
  
• for the test of 1280x720p30 create a 1280x720p30 reference video by downscaling the original reference video to 1280x720p30 with ffmpeg. Output codec again lossless (video codec utvideo, same as OBS uses)
  
• pick 2 reference frames, one from near the start of the video and one from near the end. Record the frame numbers.
• in OBS, create a new scene with one vlc video source and add the 1280x720p30 reference video
• In OBS, set base and output resolution to 1280x720 and frame rate to 30
• now create a bunch of recordings of the reference video by permutating every output setting. A autohotkey script was used for this.

Perform the next steps with each video:

• search for the 2 reference frames and record the frame numbers. This is done with ffmpeg and the psnr filter.
  
• to see by how many frames the recorded video is shifted in relation to the reference video, get the difference between the frame numbers of the high motion reference frame. For example, if the reference frame is number 2400 in the reference video but 2420 in the recorded video, the recorded video is offset by 20 frames and any frame-by-frame comparison must start these 20 frames later.
  
• compare the reference video and the recorded video and take the offset into account. "compare" means use the psnr filter of ffmpeg. Save the psnr statistics file. This way, each original frame is compared with its corresponding encoded frame and we determine the mse ("mean square error") of each frame. The lower the mse, the better the encoded frame. The better the mse of all frames of a video, the better is the quality of a video.

After processing of all videos:

• now we have all mse comparisons. For each video, we have a list of the mse of all frames of this video.
• To order our videos by quality, compare the mse list of each video with the mse list of each other video.
  
• a comparison of 2 videos looks like this: for each pair of frames (one frame from video 1 and one frame from video 2), look which has lower mse. Count the frames which have lower mse. The video that has more frames with lower mse is declared "better" than the other video.
  
• by using the this comparison method, it is possible to sort all videos by some sort of visual quality. It doesn't sort by average mse of each video but by answering the question: "which video has more frames that are better than the rest of the videos?"

***​

I also made recordings with advanced settings. But the analysis is extremely difficult to interpret, because videos with varying encoding quality (x264 with crf, Quicksync with icq) inherently score worse than videos with constant quality (CQP with hardware encoders), although they don't look worse to the eyes of a human being.

But even with bitrate-based streams, the comparison method can only be a rough estimate. The true judge can only be the human eye. For example, in another thread, I falsely claimed NVENC is better than x264, because the mse comparison said this (I apologize, @Harold). For quality-based recordings it is even more difficult - I am really lost here.

I would like to make a guide/resource from this, but how extensive should that be? It could be as short as "x264 > nvenc > quicksync" and as long as this post.

 

[Beardedbob]

Streaming 720p @ 30Fps , 2000 bitrate, CBR, Software x264 encoder.
Record 1080p @ 60fps, CQP, Quliaty 20, High Q Profile, Nvenc encoder.

Note: if you have 2.2mb upwards upload on your internet you can increase the bitrate but twitch recommends 2000 as the sweet spot atm. If you want to stream 720p 60fps then you need a min of 2500 bitrate on high motion even then it could do with higher bitrate.

 

[Wakky]

Nice work you did there even if neither MSE or PSNR can judge image quality/fidelity like our eye does.

I dont really see how you could make a guide out of this by if you do you should.

Btw did you try to really tune x264 with custom settings (me, merange, direct, etc)?

If you want to stream 720p 60fps then you need a min of 2500 bitrate on high motion even then it could do with higher bitrate.

No, no and no. Why people keep trying to stream 720p 60fps with under 4000+kbps? Where did that myth came from? You even said it yourselves; "high motion games".

Simply do the test, compare your stream at 720p 30fps and 720p 60fps for 2500kbps for both and watch how much damage you did to the image.

It's simple 720p 30fps need at least 2000 kbps to look good. You want to double the fps? Double that bitrate.


Edit: Forgot to finish before posting.

 

[Beardedbob]

Nice work you did there even if neither MSE or PSNR can judge image quality/fidelity like our eye does.



No, no and no. Why people keep trying to stream 720p 60fps with under 4000+kbps? Where did that myth came from? You even said it yourselves; "high motion games".

Simply do the test, compare your stream at 720p 30fps and 720p 60fps for 2500kbps for both and watch how much damage you did to the image.

It's simple 720p 30fps need at least 2000 kbps to look good. You want to double the fps? Double that bitrate.

Yes, yes yes........4000+ lol, really Wakky, what are you after perfection? given 4000+ requires you to be a partner on twitch then this is going to be limited. if you must use the max bandwidth you can as a non streamer then use 3500 this will limit the users who can view it given they require 3.5mb download +.

720p 30fps at 2000 isn't flawlessm in fact no even close, its down to what you want to accept as the streamer / viewer along with what game you are playing.

The one thing that's certain is there is always a trade off whether its bandwidth, CPU or quality.

 

[Wakky]

Yes I know 3500kbps is the limit for non partner but that doesnt change anything and what I say still true.

720p 60fps at 2500kbps for high motion game isn't an advice you can give that's all. It's incredible how you can say that and in your next answer you say "720p 30fps at 2000kbps isnt flawless". You were ok to go 720p 60fps 2500kbps one post before!

720p 60fps 4000kbps isn't perfection it's a minimum it makes a huge difference.

I strongly recommand you to check https://obsproject.com/forum/resources/twitch-and-youtube-settings-guide.169/ to understand how you have to chose your bitrate, resolution and fps. Your actual method if there is any isn't accurate if it can make you believe you can go 720p 60fps 2500kbps.

Yes it's all about trade of but you seems to ignore a lot of key points.

 

[Beardedbob]

Yes I know 3500kbps is the limit for non partner but that doesnt change anything and what I say still true.

720p 60fps at 2500kbps for high motion game isn't an advice you can give that's all. It's incredible how you can say that and in your next answer you say "720p 30fps at 2000kbps isnt flawless". You were ok to go 720p 60fps 2500kbps one post before!

720p 60fps 4000kbps isn't perfection it's a minimum it makes a huge difference.

I strongly recommand you to check https://obsproject.com/forum/resources/twitch-and-youtube-settings-guide.169/ to understand how you have to chose your bitrate, resolution and fps. Your actual method if there is any isn't accurate if it can make you believe you can go 720p 60fps 2500kbps.

Yes it's all about trade of but you seems to ignore a lot of key points.

As i said the min you want to work with if you want to even start to look at 720@60fps is 2500 bitrate. No one said it was going to be prefect or even outstanding quality its a start for low motion video. As stated its a trade off, pushing the bitrate to 3500 is going to limit the audience.

Thanks for the article I've already read it, though i'm not sure you have as it doesn't state 4000+. If you are partnered and have the upload then why would you stick to 720p? you might as well go up to 1080p.

As for ignoring the key points, the only thing being ignored in your fantasy world is most streamers don't have unlimited bandwidth nor are they partnered to twitch. The general Joe blogs can't stream above 2000 either due to bandwidth, pc limitations or the viewers watching can't handle the buffer pain due to their poor DL.

If you can run at 3500 bitrate its assuming you don't burst over the 3500 limited for the twitch ingress, which at CRB 3500 happens all the time resulting in twitch blocking you. This therefore brings the bitrate down even further for non partners to somewhere around 3100 to 3200 bitrate available.

 

[Wakky]

Like I said the min is way higher then that. 4000+kbps. At 2500kbps you have absolutely no reason to go higher then 720p 30fps except if you want to damage the render quality and why would anyone do that?

Pushing the bitrate wont limit the audience since Twitch give transcoding with less then 10 viewers now. I admit it was true less then one month ago but it's not anymore. I usually tell people to go 2500kbps max until they got transcoding, now I dont have to since almost everyone got it.

The article said something like 5500kbps to start to have perfect fidelity. Under 4000kbps 720p 60fps is a mess and each time you will move it will be heavily blurry. This article was more to explain you how wrong you were when you advice 720p 60fps for 2500kbps.

Why 720p and not 1080p? For high motion games the answer is the 60 fps. Specially if you consider that a lot of people wont go fullscreen to keep the chat visible meaning their display resolution is often 720p.

My fantasy world is the one of video stream and quality rendering sorry if you want to keep thinking it's a fantasy world but dont be surprised when you will realize how much damage you were or you are doing to your stream or you are doing to the stream of other and thats the worst part! Obviously you could ignore the truth but now you have no excuses, you cant give wrong advice to people anymore!
For your own stream it's your choice you can even completly fuck up the settings go 1080p 60fps for 2000kbps if you want to, its up to you but dont advice 720p 60fps 2500kbps for other please! Like never again!
In my fantasy world we like quality and we know how to reach it almost no matter the bitrate and we dont advice bad settings.

2500kbps? 720p 30fps for high motion games or 616p 45fps. Period.

Do you realize how many non partnered streamer use 3500kbps and won't never be block? Specially now with transcoding for almost everyone. Actually it's completely out of the subject. I don't care what are you reason to chose 1500kbps 2500kbps 3500kbps or whatever that wasnt the problem why the hell you deviate that much?


Now you clearly won't agree even with all the evidence of the world and you will probably go out of the subject more and more so let's stop this here and not pollute OP's thread anymore.

 

[koala]

Should I add 4000 @60fps to the analysis and provide corresponding demo videos? I thought that this high a rate is not relevant for the majority of hobby streamers, so I skipped that.

 

[Wakky]

I made a graph to know wich resolution and fps to chose depending on your bitrate there: http://imgur.com/noBGkeT

If you want to do some test for higher bitrate but yeah I believe it won't be usefull because of the 3500kbps cap for non partnered streamer. Who knows perhaps some partener will lurk around.

 

[TryHD]

I find it funny how all go for that 2000 kbit/s for non partnered is max. I stream at 5000 kbit/s with 20-70 viewers as non partnered without transcoding from twitch just fine and nobody complains about lag.

 

[Suslik V]

I compared the files 1280x720@30:

1. simple - stream - x264 bitrate=2000 preset=veryfast.mp4
2. simple - stream - NVENC_Pascal bitrate=2000 preset=highquality.mp4
3. simple - stream - NVENC_Kepler bitrate=2000 preset=highquality.mp4
4. simple - stream - QSV_Skylake bitrate=2000 preset=balanced.mp4
5. simple - stream - QSV_IvyBridge bitrate=2000 preset=balanced.mp4

IMHO, the rating (from best to worst) is:

• NVENC_Pascal (2) best
• NVENC_Kepler & x264 (3 & 1)
• QSV_IvyBridge (4)
• QSV_Skylake (5)

Why?

x264 & NVENC_Kepler - I didn't noticed any significant difference.
QSV_IvyBridge "feels" smoother at clouds movement (timeline from 00:00:47 to 00:01:00 - flying hero) than QSV_Skylake.

• 1280x720 because of my display resolution (fullscreen only 1:1 size for compare);
• 30 fps - because I watch most content in 30 fps
  Spoiler

  hope original renderer will render 30fps lossless movie for compare, not 60 to 30 convert - for synthetic tests - just to be sure how quality differ if you play at 60 fps and stream it at 30 fps, instead of 30 fps play and 30 fps stream.
• 2000kbit/s - is still too much, but theoretically, without sound, it good enough for my inet connection.
• x264 veryfast - because lower profiles has more impact on the system and rare in use (not all people have high-end CPUs :)
• All rating based only on my own perception of the footage (no numbers or math), same player for all videos, in player - default video renderer, decoder is LAV Video Decoder: 0.68.1.31-git - no hw acceleration.

 

[Suslik V]

And where is 1280x720@30 input lossless?
Maybe @Xaymar be able to record it for me at 2000kbit/s with AMD Encoder and then I'll be able to compare it visually too? At least, he has AMD card with VCE. Or I'll be wait when you'll find a friend with modern AMD card on board.

 

[xKlokwerkz]

Quick question, is there a reason to not using the High Quality preset instead of Balanced on QS? As of now, I am currently streaming using 720p60fps QS HQ and seeing this made me wonder.

 

[Xaymar]

And where is 1280x720@30 input lossless?
Maybe @Xaymar be able to record it for me at 2000kbit/s with AMD Encoder and then I'll be able to compare it visually too? At least, he has AMD card with VCE. Or I'll be wait when you'll find a friend with modern AMD card on board.

I currently have a card with broken CBR/VBR, I'm waiting for the AMD team to reply on what is causing it. I can test with a RX 4xx card once it arrives here.

 

[Beardedbob]

I find it funny how all go for that 2000 kbit/s for non partnered is max. I stream at 5000 kbit/s with 20-70 viewers as non partnered without transcoding from twitch just fine and nobody complains about lag.

The max for non-partnered is 3500 not 2000. Twitch awhile back said that the best compromise for quality vs viewers was 2000 as it game reasonable quality but also allowed for lower bandwidth users to avoid buffering.

 

[koala]

I am just uploading the 3 lossless master videos to the "lossless master" subdirectories in the corresponding directories, in case someone wants to make own recording tests. They are large, so they are probably ready this evening. Please, only download them if you really intend to test for yourself.

To reproduce my recording setting as closely as possible:

• put the lossless master video on a ssd
  
• in the video settings tab of OBS, set base and output resolution to the resolution of the lossless master video, and set the frame rate to the same frame rate as the lossless master video. Don't use the 60 fps video for recording tests of 30 fps and vice versa - only use the video with the corresponding fps and resolution.
• create a vlc media source in OBS, set its Visibility Behaviour to: "Stop when not visible, restart when visible"
• add the lossless master as only item to the playlist of the vlc media source
• deactivate the vlc media source in the main window of OBS (click on the "eye") to make it not running
• create a hotkey for the action "Show 'vlc media source'". I used CTRL-ALT-V. Set the same hotkey to "Hide 'vlc media sourc'"
• create a hotkey for the action "Start recording". I used CTRL-ALT-` (that's the backtick char). Set the same hotkey for "stop recording"
• in the stream settings of OBS, set Twitch as service. Leave the stream key empty, if you don't have a Twitch account. Although you will not actually stream to Twitch, this setting will set some encoder settings for the simple output method like keyframe interval.

Verify that if you start the vlc media source, the master video will start showing from the beginning and fill the entire canvas. If you stop it, the canvas must be empty. If you start it again, it must start from the beginning again, and if you stop it again, the canvas must be empty again.

Perform the recording this way:

• set the desired recording settings in the output tab of OBS settings and close the settings window.
  
• press the hotkey for showing the VLC media source (CTRL-ALT-V). This starts the video.
  
• exactly one second later, press the hotkey for start recording (CTRL-ALT-`). This starts the recording.
  
• wait 128 seconds
• press the hotkey for stop recording or click "stop recording"
• press the hotkey for hiding the VLC media source or click on the "eye" in the OBS GUI.

The 1 second delay between start of video and start of recording was made to prevent Windows-induced lags due to starting a bandwith-intensive file-reading situation. This way, caching and buffering can accomodate this.

I also upload the autohotkey script that automates the recordings, but beware: it's a monster. It requires 100% screen scaling of Windows (usually, the default for desktops). It defaults to the encoder capabilites of a i7-6700k with GTX 1070. If you try to record on a different hardware, you have to accommodate that in the SetGlobalConf function. The Quicksync options differ from iGPU to iGPU version.

If you try to record AMD-VCE, you must define the dialog element postions and drop-down list contents, otherwise the script will not know what and where to click. I can help with that and do it for you if you provide unscaled screenshots of the dialogs and contents of all AMD-VCE-related dropdown lists. Or an unscaled video where you click through all of these dialogs and settings, as this might be easier to create and upload than 10 screenshots. The size of the settings window must be 997x706 according to the "Active Window Info" app of Autohotkey. As far as I know, this is the OBS default. Or almost. Important is that you didn't mess with the width too much.

 

[koala]

Quick question, is there a reason to not using the High Quality preset instead of Balanced on QS? As of now, I am currently streaming using 720p60fps QS HQ and seeing this made me wonder.

I did not see any difference in the videos and the automated comparison said "Balanced" will result in better quality than high quality for most bitrates. So I chose balanced as best setting. If you want to see the complete automated rating, look for the analysis2.txt files in the 3 subdirectories. There you see each setting combined. The difference is not significant, probably not visible at all, so if you feel you should use high quality instead of balanced, go for it.

 

[Suslik V]

Anyway, please provide us with the recordings of High Quality preset Quick Sync (that I can compare it visually too, 2000kbit/s, 1280x720@30fps).

 

[koala]

I uploaded the qsv_skylake versions of quality (and speed as well).

 

[Suslik V]

I tested the last files,

...
6. simple - stream - QSV_Skylake bitrate=2000 preset=quality.mp4
7. simple - stream - QSV_Skylake bitrate=2000 preset=speed.mp4
​

Now the rating is the same as previous:

• NVENC_Pascal (2) best
• NVENC_Kepler & x264 (3 & 1)
• QSV_IvyBridge (4)
• QSV_Skylake (6 & 5)

About quality.
QSV Skylake Quality preset is better (image is clear) than Speed preset. Easy noticeable on camera move (hero landing, timeline from 00:01:03 to 00:01:18).
I didn't find difference between QSV Skylake Quality preset and Balanced preset from the footage. But it "feels" like a bit crisper image (and there, this is better) for Quality preset when camera moves around the hero (timeline from 00:00:29 to 00:00:33).
So, I can recommend Quality preset for QSV and Skylake CPUs (and don't use Speed preset for 1280x720@30 2000kbit/s).

@koala Just to be clear, the results may depend on video adapter model as well as on driver version. So, please, post HD graphics driver version and NVIDIA driver version you made tests with.

Is Quick Sync Quality preset available for IvyBridge CPUs?