Question / Help Switching from NVENC to x264 endcoding (Stream & local recording) - Best settings?

[c0nsecro]

Hello Guys,

I've upgraded my PC to the new Skylake X 2066 platform and want to switch from NVENC to x264 encoding, because x264 performs a better quality (as I heard). Now I'm looking for the "perfect" settings to get the best quality which stands in common with the bitrate, so everybody can watch my stream without buffering.

My new Specs:
CPU: i9 7900X
MB: X299 MSI Gaming Pro Carbon
RAM: 32 GB G-Skill Ripjaws 4
OS: Windows 10 (64-Bit)
Graphics: KFA GTX 1080

What I used before:
In the case ago, my CPU was the bottleneck, so I used NVENV to record local and stream my gameplay to restream.io (YouTube & Twitch). Because I'm not at home atm, I post my settings in the textform here:

Old NVENC streaming settings:
checkmark enalbled for enforce streaming service encoder settings
Encoder: NVENC
Bitrate: 6000
Keyframes: 2
Rate Control: CBR
Checkmark enabled for use 2-pass encoding
Preset: High Quality, Low Latency
Profile: high
Level: auto
B-frames: 2

Old NVENC record settings:
Encoder: NVENC
Bitrate: 25000
Keyframes: 2
Rate Control: CBR
Preset: High Quality, Low Latency
Profile: high
Level: auto
B-frames: 2

Renderer for both jobs was Direct 3D 11 and prozess priority normal.

Video settings:
Base & output resolution: 1920*1080
downscaling filter: Lanczos (sharpened scaling, 32 samples)
FPS: 59,94

I hope I get all facts in there. So now I tried to experiment with the x264-encoder settings to find out, how much my CPU can handle before the encoding start to lag. As recorded gameplay I used Watch_Dogs 2 because it handle multiple CPU-cores and is quiet actual. The max x264-settings I can use without encoding warning are as follows:

Encoder: x264
Bitrate: 6000
Keyframes: 2
Rate Control: CBR
CPU preset: medium
Profile: high
Tune: none

I just tested local recording with the bitrate I want to use for streaming and the results with x264 don't make me happy at the end. In fast actions, there are little kinds of blurring/artifacts, also there is a little border from artifacts round about the gamecharakter in the middle (realy not much, but in 1080p you can see it if you take a closer look). So I want to ask you, if you got settings, that give me a better output (additional x264 options for the command line too) based on experience you made or you use for your own? Maybe I don't need that high bitrate also, because I figured out, that not everybody can possibly watch my stream without buffering. Too much bitrate can be a problem here and so I'm no twitch-partner, nobody is able to switch between video options.

So for a better understanding, I told you, what I've planned to do:

I want to use my CPU to stream (720p/60 FPS) and record local (1080p/60FPS) if that will work. But when I'm right into it, it's a stack of two encoding actions, so I can't test an encoding with one job because the second one will be handled as an seperate action too. I hope you know what I mean ... I get two encoding prozesses with the same/different settings and my cpu will die if it try to handle it both with the settings I tested with one job. Or will OBS use the "encoded material" I stream and use it to save it local? With NVENC I can't realy figured out, because I got a drop of 20-30 FPS if I start streaming/recording. At the moment when I use the CPU, that not kinda happen because it's strong enought to handle the game AND the encoding. I know that there is a checkmark where you can say, that OBS should save the stream local too, but I don't know if the video-format was limited to .flv and that I can't edit in my video-editing-software. Also streaming with NVENC with a bitrate of 6000 produced a to bad quality I don't wanted to upload it as gameplay, so the option to save the stream was not a option until now.

When you say that OBS handle every job (streaming & recording) separately and I need theoreticaly the performance twice, I consider to use x264 encoding to stream and NVENC to record (but then I got FPS drops again) :( .

Thanks that you read my long potato-post :)
c0nsecro

 

[BK-Morpheus]

Sorry, I don't know If OBS can stream and record simultaneously with different settings, but you said you want your viewers to be able to watch your stream without buffering and you use 6000kbit/s and High Profile.
A few months back, Twitch suggested to not exceed 3500kbit/s for non partners, but right now you are allowed to use higher upload rates, as far as I know.
But most people with slightly slower DSL speeds won't be able to watch your stream (unless you have >100 Viewers or are a Twitch partner, so the Twitch server transcoding will kick in and offer lower quality transcodings to the viewers) and high profile might not be compatible with some mobile devices.

 

[c0nsecro]

Yes OBS studio can stream and record with different settings, the question is, if it use the encoded material from the streamencoding for the local recording too (so the encoder should not render the whole scene again).

Yeah, I heard from my viewers that, specialy at the weekend, my stream is buffering a lot. I think the partners get the most of the bitrate at this time and the resources are limited for normal users. I thought high profile will increase the quality too? So I had to decide what is more important :S .

 

[BK-Morpheus]

It's not only a problem of allowing partners to use more bitrate, it's also a problem of viewers with ~DSL6000 or less. They won't be able to watch your stream without buffering a lot.
High profile can make use of some x264 features that will improve the encoding efficiency and therefore give slightly better quality, but viewers with mobile devices/smart TVs etc. might not be able to watch the stream.
With "main" profile and only ~4000kbit/s your stream can be watched smoothly by more viewers.

You can locally record your stream in OBS, but only with the same settings. If OBS can record with parallel with different settings, it must be encoded with those settings, so you heavily increase the CPU load.

 

[Harold]

You can locally record your stream in OBS, but only with the same settings.

You can locally record your stream with different settings if you want.

When recording and streaming with the same settings, you use one encoder.
When recording and streaming with different settings, you use two encoders.

Choosing a higher profile in the encoder settings actually alienates potential viewers (it's why streaming services recommend "Main" as the profile).

 

[c0nsecro]

When recording and streaming with the same settings, you use one encoder.
When recording and streaming with different settings, you use two encoders.

Well, that's the struggle I got. When I changed the settings for local encoding (to increase better quality) the CPU need to encode two Jobs with two different workflows. Well, then I should record local in 1080p/60FPS with high bitrate (~25000) and NVENC and stream with x264 with average bitrate in 720p/60FPS (~4000) and a high cpu preset :/ .

 

[Harold]

don't use bitrate number based bitrate selection for recording.

You should record in CQP 15 nvenc 1080p60 and stream in CBR 4000-6000 x264 720p60.

 

[wdapunisher]

Correct me if I'm wrong, but NVENC can't really do CBR. Like Harold said, instead set CQP to a number that your encoder can handle. Not sure about 15. I've always heard 17-23 is a good range depending on system.

 

[Harold]

It's not that it can't do cbr, but it's that cbr is going to cause you other problems with recording.

cqp of 15 is what is used in the indistinguishable recording preset.

 

[c0nsecro]

Well I will try CQP with this value, I've heard when the values are smaller, the result is a better video quality. I've used CBR until now, because Adobe Premiere got problems with VBR (the audio dropping out of synch) and I was not sure, if such problems will happen with CQP too. Do you know if a value of CQP 15 give me a good quality in FPS games too?

 

[Harold]

Adobe Premiere got problems with VBR (the audio dropping out of synch)

That problem is VFR (variable FRAME rate) not VBR.

 

[c0nsecro]

I'm pretty sure that I recorded with VBR, because VFR is only available on x264 encoding. Well, that was one year and a few of builds ago, maybe it changed in the lifecyrcle. Anyways, I will try CPU encoding for streaming on 720p/60FPS, maybe I can set a fester CPU setting then. A pitty that I can't record with CPU also. With 6000 kbit/s (which is limited by Twitch) it's not possible to stream 1080p/60FPS, otherwise I would have use for record encoding the "streaming settings" and use only one encoding job. So I need two encoding jobs, what is definitely to much load for my CPU. And I don't want to use two streaming clients to upload on Twitch and Youtube separately (so it will be possible to use 9000 kbit/s for the Youtube-upload and 6000 kbit/s for Twitch), because I get the "two encoding jobs"-problem again. Well, now I need to find out how much FPS will drop when I'm using NVENC again.

Besides that, nobody got x264-parameters which I can add in the command line of x264 encoding to get a better quality result as well? I've heard that will increase the quality a bit.

 

[Harold]

Besides that, nobody got x264-parameters which I can add in the command line of x264 encoding to get a better quality result as well? I've heard that will increase the quality a bit.

You've heard wrong.

 

[Boildown]

No recent smartphone is unable to view 720p60 on High profile. So imo, you can ignore the advice about Main and stick to High. The bitrate is a far greater concern, 6Mb/s can be rough on viewers if they're eating their data plan or have lower end download speeds.

I think you should configure the NVEnc recording and the x264 streaming separately. Tweak iteratively until you've achieved your best compromise of quality vs bitrate and that your computer can handle without missing frames. Then you know those are the max settings you can attempt when you do both at the same time. In all likelihood, you'll have to turn settings down more when you do both your recording and your streaming encodes at the same time, but you will know for a fact you can't go any higher.

The i9 7900X is a 10 core CPU with hyperthreading. Meaning that x264 will open 30 threads to encode the video. The rule of thumb is that x264 can utilize 1 thread per 40 lines of vertical resolution. So if you do the math on 720p video, that's 18 threads. x264 is wasting CPU time by opening 12 too many threads. Use a custom x264 command that says "threads=18" to fix this. Of course, its very likely that 18 isn't the "best" number, you will probably get better performance with some other value, but you'll have to determine that yourself experimentally. But I fully expect that you'll be able to get better performance if you set a threads=x command where x is some value lower than the default.

That said, you need to do the steps in order. Don't tweak your threads=x until you've determined the correct x264 preset to use. This is a fine tuning adjustment, and you haven't even coarse tuned yet.

 

[CaptaPraelium]

OP I'm a bit of a noob so please take experienced users' advice, however I found that things worked better the other way - Use NVENC for local recording since you are at a high bitrate and compressing very little (trying to keep the quality high) it puts a light load on the GPU, then use x264 for the stream since it has better quality and at low bitrates you need every quality boost you can get.
This is pretty much what Harold said (Thanks for that advice Harold, I'll try those settings!) and he seems to know his stuff. I just thought I'd offer the advice since extensive testing showed me that is the best way.

 

[c0nsecro]

don't use bitrate number based bitrate selection for recording.

You should record in CQP 15 nvenc 1080p60 and stream in CBR 4000-6000 x264 720p60.

Well Harold realy know what he's talking about. I streamed and recorded CoD WWII yesterday, so a very fast FPS-Shooter. For local recording I used 1080p/60FPS with the recommended CQP 15 and high profile. The output-quality is quiet awesome, as I watched the scenes after recording, I've got the feeling the quality is better than my ingame-quality there is no blurry area :D . Well, I know that YouTube reencode the uploaded material, for 1080p/60FPS they recommended a bitrate of 12 Mbit/s, I recorded my videos with 25 Mbit/s and encoded it in Adobe MediaEncoder with 16 Mbit/s. The files with CBR and 25 Mbit/s got a size of ~4GB for 30mins, with CQP 15 I've got ~9 GB for ~13 mins. I can handle this filesize, but do you think it's a "waste" of ressources if there is such a downscaling of the quality? I have no time in the last days to make tests, I will test it, but I'm interested in your opinion.

As quality presets for the stream I've set x264 encoding with CBR 6000 kbit/s, 720p/60FPS, CPU preset: medium and high profile. Works without problems on my CPU, no problems or dropped frames over the whole time. Here you can see the results, but I think the encoding of YouTube which is done after streaming will reduce the quality quiet more than Twitch does:

YouTube-Stream
Twitch-Stream

No recent smartphone is unable to view 720p60 on High profile. So imo, you can ignore the advice about Main and stick to High. The bitrate is a far greater concern, 6Mb/s can be rough on viewers if they're eating their data plan or have lower end download speeds.

I think you should configure the NVEnc recording and the x264 streaming separately. Tweak iteratively until you've achieved your best compromise of quality vs bitrate and that your computer can handle without missing frames. Then you know those are the max settings you can attempt when you do both at the same time. In all likelihood, you'll have to turn settings down more when you do both your recording and your streaming encodes at the same time, but you will know for a fact you can't go any higher.

The i9 7900X is a 10 core CPU with hyperthreading. Meaning that x264 will open 30 threads to encode the video. The rule of thumb is that x264 can utilize 1 thread per 40 lines of vertical resolution. So if you do the math on 720p video, that's 18 threads. x264 is wasting CPU time by opening 12 too many threads. Use a custom x264 command that says "threads=18" to fix this. Of course, its very likely that 18 isn't the "best" number, you will probably get better performance with some other value, but you'll have to determine that yourself experimentally. But I fully expect that you'll be able to get better performance if you set a threads=x command where x is some value lower than the default.

That said, you need to do the steps in order. Don't tweak your threads=x until you've determined the correct x264 preset to use. This is a fine tuning adjustment, and you haven't even coarse tuned yet.

As I've wrote in top of the thread the first run with the settings from Harold worked very well and without problems with the streaming and encoding job at the same time. I will test your settings in the next days and I'm very expectant which results I will get. But are you sure that there are 30 threads? As I remember hyperthreading will double the cores, so I've got 20 threads, aren't they? Well, independently of that, 18 threads are into that, so in this case it "doesn't matter".

 

[Harold]

I recorded my videos with 25 Mbit/s and encoded it in Adobe MediaEncoder with 16 Mbit/s.

My recommendation if you're doing that sort of process is to use handbrake set to crf 15 with the veryslow preset instead of using adobe mediaencoder. That gives you 2 sources of quality improvement.

1> Giving you the same quality target as the original recording
2> Giving you a higher quality per bitrate over the mainconcept encoder used by adobe. x264 is still currently king of quality per bitrate among h.264 capable encoders.

 

[Boildown]

But are you sure that there are 30 threads? As I remember hyperthreading will double the cores, so I've got 20 threads, aren't they? Well, independently of that, 18 threads are into that, so in this case it "doesn't matter".

x264 opens 1.5 threads per logical core. A 10 core processor with hyperthreading has 20 logical cores, so 20 x 1.5 = 30.

Remember this is the last thing you should tweak. Until you've settled on a preset, and all the rest, you don't want to mess with this. But you did ask...

Besides that, nobody got x264-parameters which I can add in the command line of x264 encoding to get a better quality result as well? I've heard that will increase the quality a bit.

 

[CaptaPraelium]

don't use bitrate number based bitrate selection for recording.

You should record in CQP 15 nvenc 1080p60 and stream in CBR 4000-6000 x264 720p60.

I hope you don't mind if I get more indepth here. The migration from shadowplay to OBS is vastly under-documented so we need all the help we can get. You seem to really know your stuff with authority.

Firstly, I've noticed that while I see a 5-10% decrease (closer to 10%) in game framerates when using NVENC (as expected the card is doing the encoding work as well as graphics), if I am to run one instance of NVENC, I might as well run two, since the performance drop running a second encoder on the card is only another 2-5% more. Nvidia's documentation for the encoder suggests that this should be the case, outlining that the architecture will switch context between encoding tasks to make multiple encoders more efficient.

Counter to this, NVENC at the low bitrates required for streaming doesn't look so good, so this suggests that I should take your advice and use x264 for the stream, however streaming at 720p looks pretty darned bad for me as I'm gaming at 1440p so the downscale is extremely lossy. Small text and fine details are just a blur. If I downscale to/stream at 1080p instead of 720p, the image is far better, but even with 50% of my CPU (5820K@3.9GHz, 6 core) available the load seems too heavy. I never hit 100% utilisation, I hover around 80%, but the additional load on the CPU seems to be handled poorly by windows' scheduler. My framerates were not hugely effected, however frametime variation increased a great deal. Setting affinity to specific cores/threads for my game and OBS helped this, but I feel like this is bad practice and the scheduler should be switching context appropriately and is not.

On top of these utilisation/quality issues, is diskspace. With shadowplay recording my games in 1080p, I had a 2GB file for 10 minutes of recording (I was using the 'prerecord' buffer). Given that I'm now on 1440p, I'd expect that file size to increase to about 3.5GB, but the files I'm seeing with CQP15 are significantly larger.

All up, I feel that with half of a fairly powerful CPU to spare, I should be able to make this work, but I'm not seeing a great deal of success.... Ultimately, I'd like to be able to stream or record, and hopefully do both, without significant impact to my gameplay, but no joy so far. Any advice appreciated.

 

[Harold]

(as expected the card is doing the encoding work as well as graphics)

Except the nvenc encoder is dedicated circuitry on the card and not part of the main gpu. Large performance drops indicate overloading of the WHOLE video card more often than not, and sometimes indicate problems with pci-e lane counts.

1080p 16:9 is 2,073,600 pixels per frame.
1440p 16:9 is 3,686,400 pixels per frame.
So your expectations on file size for recording is realistic based on your baseline numbers, but CQP based recordings will vary GREATLY in size depending on motion level of the content.