Question / Help GPU Resources: New NVENC vs x264

[CreepyCastleTV]

Hello,
My PC uses an 8700k cpu and 1070 gpu. I have always run OBS Studio on x264 at the 'fast' preset with OBS set to "above normal" priority. I stream 720p60 at 4500 bitrate.

OBS doesn't report 'dropped frames' in the bottom right corner (the one with the % of dropped frames) but it struggles to stay at the constant 60 when I'm streaming Apex. I definitely saw all the tweets about this and followed all recommendations (turning off game-mode for example) and it's not too bad now but still happens and I just live with it. My Apex settings are all minimum and VRAM dedication in it is 2GB (lowest setting possible). I'm surprised this is happening when OBS has above normal priority, but whatever it is what it is...

My question:
OBS 23 has this new NVENC encoder that everyone is boasting about... but isn't it safe to say it'll only help people with weak CPUs?

OBS made it clear that Apex crushes GPU and affects OBS Studio as a result, so if I'm having these issues while on x264, how can switching to NVENC help the issue? Wouldn't NVENC use more GPU than x264?

Can decrease in CPU usage on OBS somehow outweigh the increased GPU usage, resulting in a net gain for the game and stream??

Lastly, I noticed that the new NVENC quality is noticeably worse than x264 when on a 10-series card, so I'd have to up my bitrate from 4500 to something like 6000+ I assume... does anyone stream to 6k+ bitrate on Twitch as an affiliate and not have issues with viewers struggling to watch?

Thanks!

 

[Fam3mon5ster]

#1 - technically speaking yes its a option for people that do have "weak" cpu's - as we continue to progress with more demanding games and our streams get more creative and populated with scenes and such, nvidia nvenc is a good option to alleviate the stress your cpu when utilizing x264. to get a understanding of the new update what they primarily did is the frames that are being rendered from obs are no longer being sent to your pc's ram 1st but instead going directly your gpu's vram - this optimization brings down resources and is what you see on these nvidia graphs that are posted everywhere although there is still limitations - to simply put it if you max out your cpu usage or gpu usage your always gonna run into lagged or skipped frames with obs - if your chasing the 1 pc dream of streaming / recording / and playing all in one pc - we are not there just yet - Nvidia's claim to eliminate 2 pc setup is getting closer but again with limitation


#2 the issue i have with this question is that its game specific - every game is different - some games utilize more gpu others use cpu - from what i know apex uses alot of gpu usage- so if your lagging on x264 it may or may not help - only suggestion i have here is trying it out - if you know your gpu is maxing out then yes it may not be a wise decision to use nvenc as you will run into the same issue and well if your already lagging on x264 - since both encoders may be overloaded, your only option is to try turning down settings like graphics or resolutions and go from there OR maybe the cpu is simply just overloaded and leaving the gpu not doing anything because its so stressed- you should try it and do a "recording" and under recording select use stream encoder to match what you would output -- play a match and end the recording see if you see improvement in your log -

#3 another issue i have with this question is the believe of turfing cpu usage onto gpu or vise versa - this is all gonna depend all on your build you have to find the right balance - Streaming and recording is not a one click thing you have to customize your options on everything - my issue with say youtube tutorials that say "best obs encoder settings 2019" are stupid - - it may work it may not - each pc is different - for example if i had a quad core cpu say a intel 4790k and a RTX card 2080TI i very well should opt to use nvenc as my encoder cause the games i play like dota 2 are cpu dependent so i wanna alleviate all the stress i can off the cpu if ima record or stream-- if i was gonna play say something like apex thats more gpu dependent, i may wanna op for x264 -- heres what i want you to understand - it may or may not be beneficial as the cpu still needs to be able to do its thing and communicate with the gpu - the cpu used as a example is a quad core setup 4core 8threads--it may be best to use nvenc in this situation-- if i got enough cores & threads say like a ryzen 2700x with 8cores 16 threads i may opt for x264- you need balance! you may play some odd balls like Black ops 4 for instance which seems to be both a cpu and gpu hog - you also very well may run into a situation where both x264 and nvenc are good options but it may not be a obs thing it very well may be say the game your playing for example "The Witcher" series and its "uber"graphics settings thats just be messing everything up.- this is a hard question to answer as there is multiple outcomes -

#4 one thing that everyone needs to understand is that the new NVENC is not this "OMG QUALITY UPDATE HYPE" or " I THOUGH IT WAS SUPPOSE TO MATCH MEDIUM PRESET X264 HOW DO I GET IT?" that is going on - 10 series gtx cards got a quality update on there encoders years ago - people like LINUSTECHTIPS can confirm this - they also just recently confirmed that the new RTX cards got another bump in quality - this is on the hardware side !! -- OBS introduced a new quality setting that may or may not help with visual quality in v23 -- people like "speedbre4ker" dropped a video recently showing that the old vs new nvenc didnt really improve quality if not stayed the same from v22 vs v23- he does confirm the new nvenc is better optimized though -- this whole update is performance optimization which took myself a awhile to understand --i was one of those OMG wheRe My QUaLitY uP date? guys posting-- I can stream off SLOBS and get the same quality as it looks like that obs studio v23 on new nvenc - why is this? cause the quality bump is on the hardware itself - benefits for me although if i was to come to obs studio for the new update is 1 for the optimization that all Nvidia series got and 2 may get additional quality buffs with say the psyco visual tuning option - speedbre4ker video though shows in his opinion that it didnt really help and showed a comparison as far as quality settings** im patiently waiting for other videos with comparisons hopefully "hardware unboxed" does something or gamersnexus -- anyways to not fully shoot down everyone with this quality question and with understanding of the 1st question i answer about this new update biggest achievement technically speaking if nvenc is running better more optimized for example streaming and playing net me say 90% usage - while under new nvenc net me 80 % usage now - which i can modify my obs settings maybe upping say "b-frames" or use "psyco visual tuning" which should *** give a slight increase in quality and bringing say my usage back up to say 90% ? get it? obs v23= performance with a hint of quality ** if you can afford it

but to wrap up that question cause i went off alittle lol, your 4500 bit rate may need a bump in bit rate if running at 10 series and under - us rtx users may be able to leave it the same - you dont need to be a affiliate or partner anymore to stream at 6k bit rate. but you are correct streaming at a higher bit rate may leave some potiental viewers looking at a buffering screen until you get partnered and get transcoding so that people can opt for a lower resolution. hopefully u have a steady flow of viewers if so some affiliates may get transcoding - I myself am a affliate (not a big channel and honestly ive been lacking im back to like the 1 viewer crew LUL) but i notice after you get like say 4-5 viewers you may have the possibility of getting transcoding thus i may wanna run a higher bit rate to give my viewers alittle bit more of a option but its not a guarantee - one day to another is different but thats what ive noticed. so if your looking to grow - lower bitrate is still the way to go - most people still have 720p screens i saw on a survey from steam -and a average of 2-4mb download which sounds crazy so think of that -

 

[Agamemnus]

Hey mate,

I must have said this 50 times on this forum alone by now LOL. Apex is notorious for capping out your GPU. OBS needs to do 2 things, first, it needs to draw your scene up. That might just be the game, it might include a webcam or some notifications and stuff. Whatever it is, it needs to draw it into a frame image, and then send that frame to the encoder for the encoder to actually turn the image into H.264 and send it to Twitch.

For OBS to draw the frames in your scene, it uses GPU power. This is called rendering. There's 3 types of missed frames in OBS.
1 - Lagged render frames where OBS couldn't draw up the scene in time.
2 - Skipped encoder frames, where the encoder couldn't encode the frames in time.
3 - Dropped network frames, where your net speed couldn't send fast enough (or Twitch couldn't receive fast enough).

The most common problem with Apex Legends, is #1. Your encoder is probably not at fault. So x264 or NVENC wont matter, they're both encoders and they are both doing just fine. OBS needs just a few % of your GPU's power to render/draw the scene. The catch is, that Apex will happily ask your GPU for 100% of it's power to draw frames in-game, and will fight OBS for every last %. This is where most people get their render lag. The trick is to find a way to stop Apex from demanding so much. Some people have success with v-sync, some don't. Some people manage it by lowering their graphics settings in-game some don't. Your situation will vary.

You can confirm this by attaching a log.

Another catch is that the "new" method of using NVENC in v23 of OBS has the ability to use CUDA to enhance the encode, which just requires more of your GPU. NVENC on it's own doesn't require the same processing units that the game uses, so NVENC isn't the problem. But CUDA can be for sure. So in your situation it may be a bad idea.

Yes, when you do fix the render lag, then you're right that NVENC on your GPU is worse than x264. It's not a lot worse, but it is a little.
https://unrealaussies.com/tech/nvenc-x264-quicksync-qsv-vp9-av1/

 

[Fam3mon5ster]

Hey mate,

I must have said this 50 times on this forum alone by now LOL. Apex is notorious for capping out your GPU. OBS needs to do 2 things, first, it needs to draw your scene up. That might just be the game, it might include a webcam or some notifications and stuff. Whatever it is, it needs to draw it into a frame image, and then send that frame to the encoder for the encoder to actually turn the image into H.264 and send it to Twitch.

For OBS to draw the frames in your scene, it uses GPU power. This is called rendering. There's 3 types of missed frames in OBS.
1 - Lagged render frames where OBS couldn't draw up the scene in time.
2 - Skipped encoder frames, where the encoder couldn't encode the frames in time.
3 - Dropped network frames, where your net speed couldn't send fast enough (or Twitch couldn't receive fast enough).

The most common problem with Apex Legends, is #1. Your encoder is probably not at fault. So x264 or NVENC wont matter, they're both encoders and they are both doing just fine. OBS needs just a few % of your GPU's power to render/draw the scene. The catch is, that Apex will happily ask your GPU for 100% of it's power to draw frames in-game, and will fight OBS for every last %. This is where most people get their render lag. The trick is to find a way to stop Apex from demanding so much. Some people have success with v-sync, some don't. Some people manage it by lowering their graphics settings in-game some don't. Your situation will vary.

You can confirm this by attaching a log.

Another catch is that the "new" method of using NVENC in v23 of OBS has the ability to use CUDA to enhance the encode, which just requires more of your GPU. NVENC on it's own doesn't require the same processing units that the game uses, so NVENC isn't the problem. But CUDA can be for sure. So in your situation it may be a bad idea.

Yes, when you do fix the render lag, then you're right that NVENC on your GPU is worse than x264. It's not a lot worse, but it is a little.
https://unrealaussies.com/tech/nvenc-x264-quicksync-qsv-vp9-av1/

Hey @Agamemnus question is CUDA used with the old nvenc? do you know? Im still trying to wrap my head around why the old nvenc performs better even at maxed out gpu usage - I posted some logs the other day on another post on how I found out certain scenes and or settings or even normal day to day programs we use like firefox or chrome can effect the new process - I've been messing around for about two days now and just last night I figured let me run the old nvenc under stress and see if i cant get it to tank - to my surprise I ran destiny 2 and also my gears of war 4 benchmark with the 2 insane settings bringing my 2080 ti to its knees and surprisingly the encoder did not report any issues at all. 100 % gpu bound- also did a screenshot comparison I found out that the old nvenc for me not only rendered a better picture but performed better? I'm starting to think it may be better for certain set ups to actually use old nvenc and utilize system resources vs bypassing and going directly to the gpu.

 

[Agamemnus]

Just so we don't mix our words, NVENC is the thing inside your graphics card, but the dropdown options in OBS for NVENC are OBS options.

To the best of my understanding:

The OBS option for old NVENC does not use CUDA at all to the best of my knowledge.

The OBS option for new NVENC does use CUDA if you use preset "max quality" or if you use "look-ahead" or "psycho visual tuning". If you use the "quality" preset without look-ahead or psycho visual tuning then it does not use CUDA at all.

It's still possible that the different old/new NVENC options in OBS have a different level of "success" when fighting over the % of GPU that they need. So if the game is maxing out, one MIGHT still perform better than the other, even if they're not using CUDA. On my computer, this isn't a problem at all, the new nvenc OBS option with preset quality and neither of those 2 other boxes ticked works totally fine for me on any game or scene setup I have. But that doesn't mean I can promise it will be the same for you.

All of the 3 different settings that use CUDA will increase the likelihood that you'll get render lagged frames but increase the quality you get for a given bitrate IF YOU DON'T get any lagged frames. Finding the right combination for YOU is important, when you find the settings that don't lose you any frames, you've got the settings that are right for you. If that sadly means you can't get the most quality for your bitrate, that's just unfortunate.

The whole "battle" over the GPU that OBS has with your game is for the most part just an unfortunate consequence of Windows not allowing us to give each program a certain level of priority. If you could somehow set Windows to give one application (OBS) 10% of your GPU and then let the rest of your apps (and games) fight over the remaining 90%, then this would all be easy. I don't know of a way to do this at the moment :(

 

[Mark Weiss]

I just installed 23.0.1 and tried the new NVENC option, expecting that it would reduce GPU usage as claimed. It did not. Old NVENC was using 8-9% GPU and new NVENC uses 28% GPU. My GPU is a 1060Ti. CPU is i7-9700K. Is there some secret settings that must be tweaked to gain this claimed lower GPU usage, or is it even worth bothering? Old NVENC works fine.
BTW, never could get Intel QuickSync to work as an option. OBS terminates without warning every time I try to stream with that Intel GPU mode.

 

[CreepyCastleTV]

I'd just like to say to everyone - thank you SO much for the information. I'm really no PC expert and with all the searching/youtube watching I couldn't seem to get a grasp on it. These answers are super helpful and I've already learned more in this post than the hours before. Thank you for taking the time; I really appreciate it.
@Fam3mon5ster
@Agamemnus

 

[wmy3044]

I just installed 23.0.1 and tried the new NVENC option, expecting that it would reduce GPU usage as claimed. It did not. Old NVENC was using 8-9% GPU and new NVENC uses 28% GPU. My GPU is a 1060Ti. CPU is i7-9700K. Is there some secret settings that must be tweaked to gain this claimed lower GPU usage, or is it even worth bothering? Old NVENC works fine.
BTW, never could get Intel QuickSync to work as an option. OBS terminates without warning every time I try to stream with that Intel GPU mode.

you need intel gpu driver 25.20.100.6373,it can work.

 

[Baltho82]

I read all what u said guys I have a suggestion what about he use a second gpu to encoding only just plug the monitor who use obs

 

[POPETIN]

I read all what u said guys I have a suggestion what about he use a second gpu to encoding only just plug the monitor who use obs

I tried this with a GTX 1080 Ti as primary GPU and a GTX 1050 Ti as encoding GPU, it worked great the only problem that you can encounter is that if your motherboard does not have dual x16 pcie slots running at x16 for both GPUs it will slow down your primary GPU to most likely x8 pcie