Comparison of x264, NVENC, Quicksync, VCE

BlockAboots · Nov 8, 2016

Its a shame that NVENC cant compete with x264, as x264 uses a hell of a lot more CPU cycles and the NVENC.

Speaking of which, what encoding method is best for NVENC?, VBR, CQP, or CBR??

Xaymar · Nov 8, 2016

koala said:
I also upload the autohotkey script that automates the recordings, but beware: it's a monster. It requires 100% screen scaling of Windows (usually, the default for desktops). It defaults to the encoder capabilites of a i7-6700k with GTX 1070. If you try to record on a different hardware, you have to accommodate that in the SetGlobalConf function. The Quicksync options differ from iGPU to iGPU version.

If you try to record AMD-VCE, you must define the dialog element postions and drop-down list contents, otherwise the script will not know what and where to click. I can help with that and do it for you if you provide unscaled screenshots of the dialogs and contents of all AMD-VCE-related dropdown lists. Or an unscaled video where you click through all of these dialogs and settings, as this might be easier to create and upload than 10 screenshots. The size of the settings window must be 997x706 according to the "Active Window Info" app of Autohotkey. As far as I know, this is the OBS default. Or almost. Important is that you didn't mess with the width too much.

I have no idea how to edit your scripts so I'm going to do things manually. Downloading the lossless files now (dear god that download speed), so hopefully I should have files for you in a bit.

Xaymar · Nov 8, 2016

Edit: For an odd reason, these files all have two audio streams and are below the target bitrate. Will reupload fixed one.

1920x1080: Lossless file is missing, pls upload that one too.

I discovered a bug in the currently public version while testing these and used a fixed version instead. The overall quality per bit is pretty decent, definitely useable for streaming IMO.

koala · Nov 8, 2016

Now all 3 lossless files are uploaded. My home upload bandwidth is not as fast as the bandwidth of my vps where the files are hosted ;-) The script is a simple ascii file - use your favorite text editor. It requires autohotkey. I will process your files with the automated rating @Xaymar, so we can see what the machine thinks about vce.

koala · Nov 8, 2016

It seems that VCE does not really meet the quality of the other two hardware encoders. Here is the machine-generated rating. This time, I left every quality setting of the other encoders instead of removing the inferior ones, so you can see what the difference is.

Edit: removed current analysis. Will redo and republish the analysis when Xaymar uploads new videos.

Xaymar · Nov 9, 2016

About what I expected, considering that there are quite some bugs left in the VCE firmware that need to be fixed first. I'm sure that with future driver upgrade we'll see AMDs encoding getting better and better, as the hardware is certainly capable of more.
My GPU (R9 285) for example is capable of encoding 1280x720 60 fps CBR 3500 Quality at 200 fps, so there is definitely room for improvement.

koala · Nov 9, 2016

Firmware update and driver? Interesting. Until now I thought hardware encoder really means hardware that cannot be changed, only with a new chip design. @Suslik V also mentioned this for Quicksync and NVENC and asked for my versions.

My driver versions for Quicksync:
Intel Skylake: 21.20.16.4534
Intel IvyBridge: 10.18.10.4276
(both are what Windows 10 installs and updates automatically after the anniversary update and later. I did not install them manually)

Nvidia Kepler: 368.39
Nvidia Pascal: 369.09
(these I installed manually. The Kepler driver is older, because this is my previous PC that I don't update regularly any more - only what comes in automatically)

BlockAboots said:
Speaking of which, what encoding method is best for NVENC?, VBR, CQP, or CBR??

It depends on what you want to do. For streaming, you have to use CBR. For recording, use CQP. I didn't publish recording comparisons, but if you use the "indistinguishable quality" or "high quality" setting of the simple output mode, you should be fine as they use CQP. For NVENC, "indistinguishable quality" is the same as CQP=16 (varies a bit with the output resolution) and "high quality" is CQP=23. Indistinguishable produces 2.6 times the file size of high. If you want something in the middle, use a cqp between 16..23. It's in the same quality range as x264, only larger files than x264.

Suslik V · Nov 9, 2016

I tested the new AMD files,
...

8. VCE1412_1280x720x30_CBR2000_Quality.mkv
9. VCE1412_1280x720x30_CBR2000_Balanced.mkv
10. VCE1412_1280x720x30_CBR2000_Speed.mkv

Now the rating is (almost the same as previous BUT NOT equal video used at compare, see comment below):

NVENC_Pascal (2) best
NVENC_Kepler & x264 (3 & 1)
QSV_IvyBridge (4)
QSV_Skylake (6 & 5)
AMD_VCE1412 (10)
AMD_VCE1412 (8 & 9)

Either AMD or @Xaymar who made this unbelievable "perfect" videos is f***ed up... Considering that informing AMD staff about this tests would be a good idea! How to sell items if the are fails so much? ^_^ Even worse, that this idea (VCE is shit for streaming) can spread across youtube very fast (due to current results and you'll be unable to make it any better soon).

SO, WHAT settings in OBS Studio and WHAT version of VLC you both had when you performed this test? Because, I see that @Xaymar 's videos are "darker"? Its absolutely clear that footage has different color setings (BT.601 vs BT.709 applied on encode). Please, upload test videos or images to OBS Studio from this link: https://obsproject.com/forum/resour...t-color-range-settings-guide-test-charts.442/
(you can set OBS Studio to 709, partial to be sure that you have identical setups), and thus we can see how each encoder encodes color (what standard is used by default internal processing of the hw encoder itself). Then I will compare you both again.

If possible, post test videos of HW encoders with color charts (at least, 10sec for color + 10sec for range) as input, use png image. I can compare it visually.

In general, AMD VCE has visible temporal degradation (keyframes are visible to me, when they are appear in video ^_^), and only Speed preset has less noticeable degradation, that's why I placed it higher than Balanced and even Quality preset. Maybe this is internal processing, but I think that is possible to improve in the future (maybe driver update required, of course).

Xaymar · Nov 9, 2016

Suslik V said:
bla

Jesus dude, could you be a little more friendly? There is no need for this hostility.

Anyway, for some reason my recorded videos actually used ~700kbit less than I set them to. As for the "darker colors", set your media player to use the correct decoding method. I was not able to reproduce any color difference or brightness difference with my decoder set to automatically convert Partial and Full Range correctly.

I'll redo my recordings and post new links once done.

Suslik V · Nov 9, 2016

I know that you can do the best. When things will be OK. I'll say to you: "Well done.".

Just upload test charts and make short recordings (1280x720 png image). As for x264, OBS Studio has full control over the colors. Not sure that you able to done this to the NVIDIA or AMD. You have encoded video in different standards and containers of the media files lack this info too. Probably, @Xaymar you are working with BT.709 and thus has true colors (even if it has a lot of blocking artifacts), I need to view test footage (charts) to confirm this.

Now I see, that comparison is not true. We need to improve the conditions.

Xaymar · Nov 9, 2016

Fixed files, including 1920x1080x30 this time. Remember that these files are in partial range, even though it says full - no idea why that happens and I hope I can fix it in the future.

EDIT: Found the bug that caused it, redoing these.

@koala I really need to figure out your AHK script. Would it help if I sent you a screenshot of the simple and advanced UI? I have next to no knowledge about AHK scripting.

@Suslik V AMF outputs Partial range files, so the encoded file looks more washed out than it should be. Normally this is automatically detected, but occasionally you need to set the player to convert from Partial to Full Range - like with the above files.[/USER]

koala · Nov 9, 2016

Regarding full and limited range and color space: my recordings are all partial and 601, because that's the OBS default. As far as I read about that topic, a difference between full and partial is negligible for streaming purposes, even for standard recordings that you intend to postprocess and finally publish as video. Color space 709 can/should/may be used for resolutions bigger than 1920x1080. For comparison purposes, I tried to left as much at the defaults as I can.

@Xaymar Easiest for you would be if you record a video of yourself clicking through all AMD-specific options of the simple and advanced output mode. Then I can figure out the dialog element positions for you. Multiple Screenshots are very tedious to handle. Simply start OBS 2 times: the first does a display capture of your session at the native resolution of your desktop (very important: no scaling). The second OBS is the one you are clicking through the options. For advanced output mode, it may be required that you choose and click through all rate control options, because the UI could change every time and adjust input fields according to the active rate control option.

If you really intend to use that script, I will write a documentation with a workflow that describes how to do automated recordings, automated analysis and automated ratings.

Suslik V · Nov 9, 2016

This "darker" I mentioned before is just brighter green and darker red. That points me to the different matrix (coefficients) was used when RGB becomes YUV and encoded (because my player simply decodes all HDs as BT.709 either it coded 709 or 601). Due to this transform, encoder has different data as input, thus you cannot compare footage you made. As far as I know, NVIDIA unable to encode NVENC as BT.709, can you proof other?

P.S. Also, some applications can make screens in different color space, because they can convert YUV to RGB.

Suslik V · Nov 9, 2016

To make perform the color test @koala , @Xaymar , please follow this steps:

Download OBSStudioNum1235_with_delay.zip (attached). Unzip.
Download All_files_Color_space+range+subsampling_test_charts.7z (All charts in one 7z file, ~21,7 MB - this is from the https://obsproject.com/forum/resour...t-color-range-settings-guide-test-charts.442/ ). Unpack.
- Set in OBS Studio:
  - Settings>Video>Base (Canvas) Resolution: 1280x720
  - Settings>Video>Output (Scaled) Resolution: 1280x720
  - Downscale Filter: Bilinear (Fastest, but blurry if scaling)
  - Settings>Advanced>YUV Color Space: 709
  - Settings>Advanced>YUV Color Range: Partial
  - Settings>Advanced>Color Format: NV12
- Save all settings (by OK button)
Setup all settings you need to the encoder.
When encoder setup is ready, add two image sources to the OBS Studio scene.
Rename upper image source (first in the list of the Sources) to 'Range chart'.
As Image File choose: 1280x720_color_range_test_chart.png file (in source Properties).
By default, image starts from top left corner of the screen (position 0;0) do not change this.
Rename second image source (second in the list of the Sources) to 'Color chart'.
As Image File choose: 1280x720_chroma_subsampling_test_chart.png file (in source Properties)
By default, image starts from top left corner of the screen (position 0;0) do not change this.
- Set in OBS Studio:
  - Settings>Hotkeys>Show 'Range chart' : Num5
  - Settings>Hotkeys>Hide 'Range chart' : Num2
  - Settings>Hotkeys>Start Recording : Num1
  - Settings>Hotkeys>Stop Recording : Num3
- Save all settings (by OK button)
Click at the center of the OBS Studio Preview window (this defocuses Sources list).
Run OBSStudioNum1235.exe (attached in zip).
Click at the center of the OBS Studio Preview window (this defocuses Sources list).
Wait for 2+10+10+2=22 seconds. The ...recording start, 'Range chart' hides, recording stop, 'Range chart' shows... sequence complete. The OBSStudioNum1235.exe should disappear from the tray.
Upload recorded files on the forum (or give me a downloadable link to them to view the results).

The AutoHotkey script itself, in case of false positive antivirus warning for you (exe+ahk in .zip attached):

Code:

Sleep, 2000
Send, {Numpad1}
Sleep, 10000
Send, {Numpad2}
Sleep, 10000
Send, {Numpad3}
Sleep, 2000
Send, {Numpad5}

In case of events not triggered, try to modify the script, see Numeric hotkeys not working on Windows OBS studio about Up and Down time for keys.

Xaymar · Nov 9, 2016

I figured out why the colors seemed off, obs doesn't actually respect what the plugin asks for in get_video_info. I made a bug report about that here.

koala · Nov 9, 2016

It seems to me that information flow should be the other way round: the plugin should use what OBS asked it to use. If the plugin isn't able to conform, it should return an error and don't record anything. If the plugin/encoder is able to override user-defined settings, the resulting video can contain something the user didn't request, and the user don't even get a notice about this. This is bad.

Ideally, you shouldn't be able to configure something in OBS an encoder isn't actually able to do. For example, if the encoder isn't able to output color format i444, which is the case for Quicksync, you shouldn't be able to set both this color format AND choose Quicksync. Another example, if you try to use a resolution larger than 4096x4096 with NVENC, this will always fail, because the maximum supported resolution of NVENC is 4096x4096.

As alternative, the "start streaming" or "start recording" button should be inactive and a message should be visible that tells the user that the settings he requested cannot be used for that encoder.

But this is not what this thread was meant for.

Suslik V · Nov 9, 2016

koala said:
...But this is not what this thread was meant for.

Maybe later I could post some BitrateViewer screens (with my thoughts about: constant "bitrate" and what it mean for NVENC, Quck Sync and software x264 from this MP4 test files; why analog TV, frequency and filler data important) or maybe not - CBR is so hard to understand and it was used in the tests of quality. For example, how you can transmit 2000kbit/s if Ethernet is 100Mbit/s itself? How this 100Mbit/s is limited? Maybe it is impossible to stream with this settings at all (CBR for Quck Sync vs CBR for NVENC) and at the same settings one site (provider/host) can accept this bitrate value and other unable - it require to lower your setting or you'll get the frame drop? If settings useless or not equal for compare than this thread goes to nowhere.

You need to say that you targeting the real applications (services) and emulate its work for this tests. Thus overall "quality" may differ from the quality numbers you calculate. The bitrate for CBR is the main restriction and you only assume that encoder controls the bitrate very precisely (and what if it fluctuates?). This is important for streaming. Maybe Service can average your bitrate for last 10 seconds (or for 0.5 sec - who knows) and drop few frames as overheated bandwidth.

koala · Nov 9, 2016

Please don't make this an universal "what to consider for streaming" thread. This whole comparison is only to answer the question what the best encoder for a given situation probably is. Considering all other external circumstances are identical: bandwidth, application, stream viewers - which encoder should I choose?

For this question, I don't need to consider anything else besides the encoder. Network or bitrate fluctuation is irrelevant, because if it fluctuates with x264, it fluctuates with NVENC as well. If my upload speed is too narrow for 2500 for x264, it is too narrow for 2500 for NVENC as well. If x264 provides better quality than NVENC, then it does this regardless of network fluctuations and lost packets. So all these external conditions are irrelevant for these tests.

For comparison, you need an equal base for every encoding test and reproducible results. This is what I try to provide. But not an emulation of the whole internet. That you cannot stream with 2500 if your local bandwidth gives only 2000 is a natural law. That your stream looks awful if you lose 30% of your frames due to too low CPU performance is a natural law - I don't need to mention this in these comparisons. And as long as streaming services require CBR, I don't need to consider any other rate control mechanisms for streaming. CBR and the bitrate fluctuation within a given time frame is precisely defined by the h264 standard and well regarded by the Internet backbone providers - I don't need to look deeper into that. I assume all encoders adhere to the standard.

Xaymar · Nov 9, 2016

Suslik V · Nov 9, 2016

@Xaymar , can you make 1280x720@30 fps 2000 Partial range videos for me (Quality, Balanced and Speed)? Because, it is full range now and may consume bitrate for this additional bits encoding. Also, I don't know how topic starter can compare extracted frames of different ranges (they were in the base).

Comparison of x264, NVENC, Quicksync, VCE

Member

Active Member

Active Member

Active Member

Active Member

Active Member

Active Member

Active Member

Active Member

Active Member

Active Member

Active Member

Active Member

Active Member

Attachments

Active Member

Active Member

Active Member

Active Member

Active Member

Active Member