NVENC Encoder Overload at Alan Wake II with RTX4090/Ryzen 9 7950X3D/64GB RAM ?

[c0nsecro]

Hello all,
hopefully you are able to help me. I've got massive problems recording/streaming Alan Wake II with NVENC enconding at OBS. I've got a lot off workload because im playing at 1440p on the same rig where I'm recording in 1440p60FPS, streaming in 1080p60FPS and using the virtual cam for TikTok-Streaming with TikTok-LiveStreaming software (1080p60FPS). I've got no problems with at least Resident Evil 4, but Alan Wake still laggs a lot and I see dropping the CPU encoding FPS in the right lower corner of OBS. The CPU is not on heavy load, the graphic card runs on heavy load (~96%) and dropped sometimes in lower percentages. The temperature is no problem because of a custom water cooling (graphic card is around 45°C).

The log-file is attached. Someone else got problems as well?

 

[Harold]

Change your encoder preset down to P1 for recording
Because you're using CQP as your rate control, quality won't suffer.

 

[c0nsecro]

Change your encoder preset down to P1 for recording
Because you're using CQP as your rate control, quality won't suffer.

Ok thank you, I will edit that. But is it possible that the load from the Encoder is too high? The RTX40th series got two Encoder chips, one reason I bought it. I thought the GPU and NVENC load is independent from each other.

 

[Harold]

The P6 and P7 presets use more than just the encoder chips for their processes, and run on both the encoder chip AND on shared components of the video card. When you're recording, the quality per bitrate gains from using a high number preset are NOT worth it.

 

[c0nsecro]

So I've tried the other settings and it doesn't help, changing the setting won't affect anything. It's lagging when I'm just streaming without recording AND it's lagging when OBS is just open. I've seen that the GPU load drops when I've got the laggs and that the GPU load is 99% the most of the time when I'm running the game without OBS. There are % drops as well, but not that hard as OBS is open. So I think the GPU is under such a high load, that the encoder gets affected as well (I've read that the encoder could struggle if the workload is higher than 96%)?

 

[Tomasz Góral]

Check on lower resolution, probably scale from 1440p to 1080p kill resources. If use 1080p you send directly frame to encoder, if use 1440p need some CUDA resources before send frame to NVENC.

 

[karlos103]

Got basically same system except 32gb of ram ...i play at 4k but record at 1440p .
Game plays perfect on my LG OLED 120hz tv , but in preview window on OBS you can see the stuttering as i pan left and right .
Very micro stutters ...dunno how to explain it lol ( like playing a game at 120hz then dropping to 60hz ..not as smooth )...but when i play say Layers of Fear ...same settings ...it smooth as butter.
Do any of the settings in OBS aftect the preview window , or just when recording ?
The Invincible also records fine.

Also have same issue with Robocop ...stutters.
Is it something to do with the game engines causing this issue ?
I have tried recording at different resolutions , tried a capture card recording to same pc ...still same.
I record to the same pc by the way.
When i watch my youtube recordings from my phone , they dont look as bad , guessing its the 120hz refresh rate of screen helping.
Im thinking its game engines causing the issue as they are new games and maybe not optimised that well yet ...but then why play fine ...only recording issue ...strange

 

[c0nsecro]

So for me I "fixed" my problem by limiting the FPS in the Nvidia control panel, so the load on the GPU isn't higher than the magical 96%. After that the encoding was fine again.

 

[c0nsecro]

Got basically same system except 32gb of ram ...i play at 4k but record at 1440p .
Game plays perfect on my LG OLED 120hz tv , but in preview window on OBS you can see the stuttering as i pan left and right .
Very micro stutters ...dunno how to explain it lol ( like playing a game at 120hz then dropping to 60hz ..not as smooth )...but when i play say Layers of Fear ...same settings ...it smooth as butter.
Do any of the settings in OBS aftect the preview window , or just when recording ?
The Invincible also records fine.

Also have same issue with Robocop ...stutters.
Is it something to do with the game engines causing this issue ?
I have tried recording at different resolutions , tried a capture card recording to same pc ...still same.
I record to the same pc by the way.
When i watch my youtube recordings from my phone , they dont look as bad , guessing its the 120hz refresh rate of screen helping.
Im thinking its game engines causing the issue as they are new games and maybe not optimised that well yet ...but then why play fine ...only recording issue ...strange

So maybe it's just the preview that stutters and the recorded files are fine, try to ignore it or better deactivate the preview in OBS. With my old 3090 I've got sometimes problems, that I've got an "Encoding overload issue" because the preview needed some resources as well. Not much, but enough to cause problems with the encoder.

 

[dreamrise1]

Did you just limit to 60 fps? I have the same system, limited to 60 but it's still "overload". Plus it crashes at some point either just obs or the whole PC. It only happens to Alan Wake :(