OBS 29.0.0 with 4090 FE @ 4k struggling to encode

[MNeverOff]

Hey all,
I've been troubleshooting this issue for the past few days, looking for every bit of help I can get. I am trying to configure a 4k streaming profile, current settings:

02:31:31.621: obs_output_set_audio_encoder: Null 'output' parameter
02:31:31.621: output 'adv_stream' (rtmp_output) created
02:31:31.661: [jim-nvenc: 'advanced_video_stream'] settings:
02:31:31.661: codec: H264
02:31:31.661: rate_control: CBR
02:31:31.661: bitrate: 10000
02:31:31.661: cqp: 20
02:31:31.661: keyint: 120
02:31:31.661: preset: p6
02:31:31.661: tuning: hq
02:31:31.661: multipass: disabled
02:31:31.661: profile: high
02:31:31.661: width: 3840
02:31:31.661: height: 2160
02:31:31.661: b-frames: 2
02:31:31.661: lookahead: true
02:31:31.661: psycho_aq: true
02:31:31.661:
02:31:31.777: ---------------------------------
02:31:31.778: [FFmpeg aac encoder: 'adv_stream_aac'] bitrate: 160, channels: 2, channel_layout: stereo
02:31:31.778:
02:31:31.781: [rtmp stream: 'adv_stream'] Connecting to RTMP URL rtmps://a.rtmps.youtube.com:443/live2...
02:31:31.782: rtmp-common.c: [open_json_file] Wrong format version (3), expected 4
02:31:31.784: rtmp-common.c: [open_json_file] Wrong format version (3), expected 4
02:31:31.808: [jim-nvenc: 'advanced_video_recording'] settings:
02:31:31.808: codec: AV1
02:31:31.808: rate_control: CBR
02:31:31.808: bitrate: 5000
02:31:31.808: cqp: 20
02:31:31.808: keyint: 120
02:31:31.808: preset: p6
02:31:31.808: tuning: hq
02:31:31.808: multipass: disabled
02:31:31.808: profile: main
02:31:31.808: width: 3840
02:31:31.808: height: 2160
02:31:31.808: b-frames: 2
02:31:31.808: lookahead: false
02:31:31.808: psycho_aq: true
02:31:31.808:
02:31:31.833: [rtmp stream: 'adv_stream'] Interface: Intel(R) I211 Gigabit Network Connection (ethernet, 1000 mbps)
02:31:31.930: ---------------------------------
02:31:31.930: [FFmpeg aac encoder: 'Track1'] bitrate: 320, channels: 2, channel_layout: stereo

Issue:
After a few minutes (can be 1 minute, can be 15 minutes, can be even 30 minutes) GPU encode seems to start lagging behind severely up to a point where every new frame is skipped due to the encoding lag, even though it's set on laughable bitrates (10k stream, 5k recording). Same is on 50k/25k accordingly, no clear dependency on the values, seems random.
This is present on both capture card passthrough from a PS5 via an Elgato 4K60 Mk.2 and on-device 4090 FE 4k gameplay. Even if it's capped to 30FPS output, even with Raytracing disabled.

On-device screenshot: https://i.imgur.com/vKFNW2Z.png, video: https://youtu.be/AWuggNhQ1TI
PS5 screenshot: https://i.imgur.com/iUfxNea.png, video: https://youtu.be/3QG9YRsBJaw
(Sorry about the crap quality, but one can clearly see encoding lag out of control and the fact that in some of these the Video Engine Load is pegged at 100% and GPU is not even at 50%)

As you can see from the videos, whenever the encoding lag starts to creep up the Task Manager view of the Encode utilization switches from very stable to spotty and ultimately stops being stable 70% but rather jumps between 50 and 70%, which I feel is the only thing other than GPU-Z GPU Load spike that indicates an encoding lag.

Spec:
- AMD 5800X3D
- 64GB of 3600 CL16 DDR4 RAM (G.Skill Trident Z)
- RTX 4090 FE
- Corsair RM750X 2018 (a weak link, yes, but don't dismiss it for it)
- Fractal Meshify 2 with 9 fans (no lack of cooling)
- RAID0 Intel 660p 2TBx2 SSDs (hardware raid, AMD board. Does not change if it's recorded to a 970 Evo 1TB direct access) for the recording destination
- Both mild overclock (98.7% TimeSpy Stress Test) AND Stock. OC for PS5, Stock for Witcher.

More logs:
- Witcher PC run:

02:31:31.958: ==== Recording Start ===============================================
02:31:31.959: [ffmpeg muxer: 'adv_file_output'] Writing file 'D:/Recordings/OBS/2023-01-14 02-31-31.mkv'...
02:31:32.267: [rtmp stream: 'adv_stream'] Connection to rtmps://a.rtmps.youtube.com:443/live2 successful
02:31:32.267: ==== Streaming Start ===============================================
02:31:32.267: [rtmp stream: 'adv_stream'] Socket send buffer is 65536 bytes
02:33:44.673: warning: EOI missing, emulating
02:37:33.200: warning: EOI missing, emulating
02:38:58.293: [ffmpeg muxer: 'adv_file_output'] Output of file 'D:/Recordings/OBS/2023-01-14 02-31-31.mkv' stopped
02:38:58.293: Output 'adv_file_output': stopping
02:38:58.293: Output 'adv_file_output': Total frames output: 26348
02:38:58.293: Output 'adv_file_output': Total drawn frames: 26777 (26780 attempted)
02:38:58.293: Output 'adv_file_output': Number of lagged frames due to rendering lag/stalls: 3 (0.0%)
02:38:58.293: ==== Recording Stop ================================================
02:38:58.409: warning: 2 frames left in the queue on closing
02:38:58.448: [rtmp stream: 'adv_stream'] User stopped the stream
02:38:58.448: [rtmp stream: 'adv_stream'] Socket send buffer is 2097152 bytes
02:38:58.448: Output 'adv_stream': stopping
02:38:58.448: Output 'adv_stream': Total frames output: 26329
02:38:58.448: Output 'adv_stream': Total drawn frames: 26797 (26800 attempted)
02:38:58.448: Output 'adv_stream': Number of lagged frames due to rendering lag/stalls: 3 (0.0%)
02:38:58.449: Video stopped, number of skipped frames due to encoding lag: 4934/26387 (18.7%)
02:38:58.449: [rtmp stream: 'adv_stream'] Freeing 1 remaining packets
02:38:58.457: ==== Streaming Stop ================================================

- Valhalla PS5 run:

23:16:23.669: ==== Recording Start ===============================================
23:16:23.669: [ffmpeg muxer: 'adv_file_output'] Writing file 'D:/Recordings/OBS/2023-01-13 23-16-23.mkv'...
23:16:24.013: [rtmp stream: 'adv_stream'] Connection to rtmps://a.rtmps.youtube.com:443/live2 successful
23:16:24.013: [rtmp stream: 'adv_stream'] Socket send buffer is 65536 bytes
23:16:24.013: ==== Streaming Start ===============================================
23:20:54.376: [ffmpeg muxer: 'adv_file_output'] Output of file 'D:/Recordings/OBS/2023-01-13 23-16-23.mkv' stopped
23:20:54.376: Output 'adv_file_output': stopping
23:20:54.376: Output 'adv_file_output': Total frames output: 15434
23:20:54.376: Output 'adv_file_output': Total drawn frames: 16242
23:20:54.376: ==== Recording Stop ================================================
23:20:54.455: warning: 2 frames left in the queue on closing
23:20:55.431: [rtmp stream: 'adv_stream'] User stopped the stream
23:20:55.431: [rtmp stream: 'adv_stream'] Socket send buffer is 2097152 bytes
23:20:55.431: Output 'adv_stream': stopping
23:20:55.431: Output 'adv_stream': Total frames output: 15473
23:20:55.431: Output 'adv_stream': Total drawn frames: 16312
23:20:55.432: Video stopped, number of skipped frames due to encoding lag: 4154/15524 (26.8%)
23:20:55.432: [rtmp stream: 'adv_stream'] Freeing 2 remaining packets
23:20:55.444: ==== Streaming Stop ================================================

 

[MNeverOff]

Update:
Tried OBS 28.1.2 with both AV1 recording and H.264 (on assumption that same codec on stream and recording would somehow help).
Still encountered the same issue around 3 minutes into the test. Log ling: https://obsproject.com/logs/iN6ihbYV4ShCc3CT

Config: 48k P6 qres 4k 60fps stream, 24k P6 qres 60fps recording. If you review the screenshot attached it's very clear when the encode drops and the encoding lag begins: https://i.imgur.com/Tut1FmB.png

 

[sandrix]

P5-P7 are not suitable for 4K encoding for H.264. P6-P7 is not suitable for H.265

There is a migration table, but these are just recommendations. In practice, it works differently.

NVENC Preset Migration Guide

docs.nvidia.com

 

[MNeverOff]

P5-P7 won't handle 4K encoding for H.264. P6-P7 fails for H.265

Hey sandrix,

thanks, that looks like a massively useful chart. Granted, it seems to be titled "turing" which, if I'm not mistaken, would correspond to a 20-series card. I'm assuming that 40-series can dish out much, much more than that?

Also, wouldn't it overload right away then, and not at a random period of time after it starts? Doesn't bitrate matter at all for this chart, is it purely a factor of resolution and preset?

 

[sandrix]

Hey sandrix,

thanks, that looks like a massively useful chart. Granted, it seems to be titled "turing" which, if I'm not mistaken, would correspond to a 20-series card. I'm assuming that 40-series can dish out much, much more than that?

Also, wouldn't it overload right away then, and not at a random period of time after it starts? Doesn't bitrate matter at all for this chart, is it purely a factor of resolution and preset?

There are certainly differences, but in general, Turing and Ampere have the same encoder for H.264 and HEVC.

Nvidia NVENC - Wikipedia

en.wikipedia.org

Video Encode and Decode GPU Support Matrix

Find the related video encoding and decoding support for all NVIDIA GPU products.

developer.nvidia.com

The easiest way to see the load is in the task manager. Personally, I use MSI Afterburner + RTSS for this

 

[sandrix]

However, you know how to monitor the load. In general, don't let the encoder reach 100%, and don't let the GPU load reach 100% as well. NVENC need CUDA resources.

Encoder Features using CUDA​

Although the core video encoder hardware on GPU is completely independent of CUDA cores or graphics engine on the GPU, following encoder features internally use CUDA for hardware acceleration.
Note: The impact of enabling these features on overall CUDA or graphics performance is minimal, and this list is provided purely for information purposes.

• Two-pass rate control modes for high quality presets
• Look-ahead
• All adaptive quantization modes
• Weighted prediction
• Encoding of RGB contents

 

[MNeverOff]

However, you know how to monitor the load. In general, don't let the encoder reach 100%, and don't let the GPU load reach 100% as well. NVENC need CUDA resources.

Encoder Features using CUDA​

Although the core video encoder hardware on GPU is completely independent of CUDA cores or graphics engine on the GPU, following encoder features internally use CUDA for hardware acceleration.
Note: The impact of enabling these features on overall CUDA or graphics performance is minimal, and this list is provided purely for information purposes.

• Two-pass rate control modes for high quality presets
• Look-ahead
• All adaptive quantization modes
• Weighted prediction
• Encoding of RGB contents

This is extremely fair and valuable and it feels like the Two-Pass QRES is something that I'm ought to disable for sure. Would you say that it also could explain why the PS5 captured gameplay is experiencing this issue? I am trying to use that in order to isolate any potential game load and prevent it from influencing the result.
Also, doesn't 4090 having two encoders change the chart in question in its favor? As in, two encoding units?

 

[MNeverOff]

I have, however, an even more interesting thing to report.
I have been experimenting further and it seems that the order in which the things are done (record, then stream vs stream then record) matters when one is using 4090 with H.264 and AV1.

In short, I was able to sustain over 15 minutes of capture card-produced P6 42k/21k stream/record stream (trimmed the unsuccessful attempts from the log): https://obsproject.com/logs/ro95OYHllLKOVuB1

Observed Vid MSI Afterburner consumption as well as the general overview of the stream.

I ended up running another test and that one started falling out with encoding lag after 17 minutes only.

 

[sandrix]

You just need to choose the right preset, for example P4. You're just pushing the encoder to the limit and only making things worse. Don't you see this in GPU-Z and MSI Afterburner?
91 missed frames out of 59518 is nothing. It becomes really a problem when more than 1-3%.

It is also desirable to disable the preview, this negatively affects performance and stability.
Most often this happens when the GPU is loaded up to 100%.

ON

OFF

 

[MNeverOff]

You just need to choose the right preset, for example P4. You're just pushing the encoder to the limit and only making things worse. Don't you see this in GPU-Z and MSI Afterburner?
91 missed frames out of 59518 is nothing. It becomes really a problem when more than 1-3%.

It is also desirable to disable the preview, this negatively affects performance and stability.
Most often this happens when the GPU is loaded up to 100%.

ON

OFF

Right, I understand the principle that lower presets will yield better results, as well as disabling the preview for the cases when the game is running natively.

What I can't understand is why I can successfully push it all the way to P6/P6 when using a capture card and it'll do fine for whole 15 minutes+ but then, seemingly out of nowhere, it will all collapse?

 

[rockbottom]

Not enough reliable power, especially during high load periods. Get a new PSU, there's no way your system is being powered correctly with the RM-750. All (3) power cables for the 4090 need to connect directly to the PSU & there doesn't appear to be enough connections to support that & the rest of your system. Look at the Performance Cap Reason in GPU-Z when under high loads, most likely voltage reliability issues.

The 970EVO is the only drive you should be writing to & even that is DRAMless so it may be an issue. The sustained write performance of the 660p's is terrible. Light duty only for those drives.

As stated above, P4 is the preset that should be used for 2160p 60FPS.

 

[Acey05]

Do you mean the video starts stuttering at intervals or something like that, and will keep on doing it, or does it happen only once?

If it happens once, it's a common issue with software "ramping" up, such as changing scenes and software priority. You can run OBS as Admin and set it to High Performance for Graphic Card usage to bypass that issue.
As others have mentioned, your Power Unit could also be conflicting (hitting the caps of a smaller PSU is more of an issue than a bigger unit) depending on the game.

If it's random, it's a Preset issue. I don't own a 4090, but unlike what many Streamers or Youtubers said, using a higher P# is not viable, depending on how you capture or what kind of games you're playing. I don't know why everyone started claiming you can run P7 on any game without performance cost.
You can also play the simplest Unity made game that looks like a PS1 game, and many games today, even if vsync capped, try to "use everything they can from you GPU", this murders OBS if any of your options require extra CUDA or Memory to be used.

If you're using H265/HEVC, you need to be careful, it's brutal, and even the lowest preset can be hard to run for some people (I never had success with it on a 1060).

If it happens at intervals, it's an annoying "Capture Issue" with OBS. Depending on the Frames you're targeting and what you're Capturing, if OBS desyncs the timing, it will start dropping Frames to try and Fix itself with the Graphics being rendered to your output.
I think it's mostly related to having a bad refresh rate between your monitor and GPU (60 FPS = 60.000240 frames, if your refresh rates are something like 59.998800, this will break your capture at intervals).
This is most apparent with Display Capture, Window Capture still has this issue but only for a couple of seconds (Game Capture doesn't have this issue).
I never figured out how to fix this issue, and it may even be a windows issue for all I know.

 

[MNeverOff]

Not enough reliable power, especially during high load periods. Get a new PSU, there's no way your system is being powered correctly with the RM-750. All (3) power cables for the 4090 need to connect directly to the PSU & there doesn't appear to be enough connections to support that & the rest of your system. Look at the Performance Cap Reason in GPU-Z when under high loads, most likely voltage reliability issues.

The 970EVO is the only drive you should be writing to & even that is DRAMless so it may be an issue. The sustained write performance of the 660p's is terrible. Light duty only for those drives.

As stated above, P4 is the preset that should be used for 2160p 60FPS.

Hey, thank you for the insight!

I would maintain that this is not a PSU problem, especially seeing as entire GPU draws under 100W when encoding a capture card stream from the PS5 as seen on screenshots a few messages up. Performance cap is indeed Voltage but it's not indicative of any PSU issues as stated in the official documentation.

660p in their almost empty state should still have no issues writing what, 5,000 Kbps bitrate, right? But yeah, anyways, recording to a 970 Evo doesn't change this at all.

If recording in CBR then indeed, P4 / P4 seems to work quite well, thanks for reinforcing that suggestion, I tried and whilst I'm not thrilled with the quality it definitely works!

 

[MNeverOff]

Do you mean the video starts stuttering at intervals or something like that, and will keep on doing it, or does it happen only once?

If it happens once, it's a common issue with software "ramping" up, such as changing scenes and software priority. You can run OBS as Admin and set it to High Performance for Graphic Card usage to bypass that issue.
As others have mentioned, your Power Unit could also be conflicting (hitting the caps of a smaller PSU is more of an issue than a bigger unit) depending on the game.

If you're using H265/HEVC, you need to be careful, it's brutal, and even the lowest preset can be hard to run for some people (I never had success with it on a 1060).

If it happens at intervals, it's an annoying "Capture Issue" with OBS. Depending on the Frames you're targeting and what you're Capturing, if OBS desyncs the timing, it will start dropping Frames to try and Fix itself with the Graphics being rendered to your output.
I think it's mostly related to having a bad refresh rate between your monitor and GPU (60 FPS = 60.000240 frames, if your refresh rates are something like 59.998800, this will break your capture at intervals).
This is most apparent with Display Capture, Window Capture still has this issue but only for a couple of seconds (Game Capture doesn't have this issue).
I never figured out how to fix this issue, and it may even be a windows issue for all I know.

Once it "falls" into the lag it's just unbearable, it encodes maybe 1 frame out of every 100, when playing the recording back it's not even a slideshow. Sadly, it's happening more than once and is somewhat stable in its randomness.

Preset is definitely what is causing this issue. I can easily stream CBR P7 51k qres at 4k60hz with the 4090 btw, but not do that and record at the same settings. Not interval based though, so most likely doesn't have anything to do with OBS.

Thanks for the whole throve of ideas, this really helps me check my bases and I appreciate the time you took out for this!

 

[MNeverOff]

Seems like I've found a temporary solution for this, courtesy of u/Williams_Gomes/ on Reddit: https://www.reddit.com/r/obs/commen.../?utm_source=share&utm_medium=web2x&context=3

Essentially, it boils down to using CQP for recording at waaay lower presets (P1-P2). This achieves great quality and low (15-16% Video Engine Load) encoder load.

In general, I feel like I understand what is closer to the cause of the issue, thanks to the MSI Afterburner charts:

When I'm trying to encode using a high preset, even with laughably low bitrates, it seems like the encoding pipeline first applies a preset to the frame and only afterwards does the bitrate somehow come into the question. So it's impossible to balance out an encoder overload due to high presets by lowering the bitrate (might be obvious to users on this forum, wasn't to me). The only solution is to find a combination of a Streaming and Recording presets that don't overload the Video Engine Load value too much and stay below 100% at all times.

I still can't explain why when encoding a stream off of a capture card with insane presets (P6/P6 4k60 48k bitrate both) if I initiated the Recording first I could get through the first 10-20 minutes with 0 issues, but if I initiated the Stream first it would die right away, but alas, not something that I'm knowldegable enough to solve at this time.

Thank you everyone for pointing me towards a solution, for now I'm happy to report that I can sustainably stream P7 qres 51k and record in CQP P1 with amazing quality on both and about 20% headroom on the encoder - both on native gameplay at 4k and when encoding a capture card stream. 4090 is indeed quite something!

 

[BB4U]

Couldn't remember my old account so had to make a new one to comment.

I have a i5 12600k 4080 build and a 9600k 3060ti build with 600/600 fiber win11 both fresh installs
before I got the 4080 my 3060 was paired with the 12600k
And before OBS 29 I was on 28/beta

I to use a elgato mk.2 but @1440p xbox series X/PC to PC cap card CBR 25k BR 3060ti YOUTUBE vp9 1440p stream key.
My P7/P6 YouTube streams ran flawless and so crispy and I mean crispy with the 3060ti OBS/28/beta

Took a break from streaming then
I upgraded to the rtx 4080 and did a fresh reformat and upgraded to OBS 29
Test I have done 25 to 35k BR CBR 1440p 60fps OBS
streams from my 4080 single PC p5/p7 capturing pc gameplay -10 to 15fps hit ( no cap card)
streams from the second PC all done through capture card with the 3060ti as a dedicated encoder PC p4/p7
and swapped for arc a380
Mid to highest quality settings for each card.
And OBS teleport ( no cap card)

Issues
none of the cards even came close to encoder max out (a380 is a monster i didnt even know it was encoding the usage was so small lol)
But now the artifacting and video quality are so poor now in every test compared to the older versions of OBS on top of weird disconnects and even what looks like inconsistent frame rates
Twice with the 4080 test I got some sort of encoder lag to.

Now I would of said ok maybe its a driver issue or bios with 4080.
But after testing all of what I did my guess is its a OBS or windows issues since both had big updates.

Just adding my two cents to this since I've been pulling my beard out trying to solve this lol. And seen you having a similar issue.

 

[Acey05]

If you go back to OBS 28, do you still have the same issue?

Otherwise, it sounds at this point like a Windows issue or something related to the one of the updates which may be conflicting with GPU Drivers and their overheads.

 

[N8Nefarious]

Hey all,
I've been troubleshooting this issue for the past few days, looking for every bit of help I can get. I am trying to configure a 4k streaming profile, current settings:


Issue:
After a few minutes (can be 1 minute, can be 15 minutes, can be even 30 minutes) GPU encode seems to start lagging behind severely up to a point where every new frame is skipped due to the encoding lag, even though it's set on laughable bitrates (10k stream, 5k recording). Same is on 50k/25k accordingly, no clear dependency on the values, seems random.
This is present on both capture card passthrough from a PS5 via an Elgato 4K60 Mk.2 and on-device 4090 FE 4k gameplay. Even if it's capped to 30FPS output, even with Raytracing disabled.

On-device screenshot: https://i.imgur.com/vKFNW2Z.png, video: https://youtu.be/AWuggNhQ1TI
PS5 screenshot: https://i.imgur.com/iUfxNea.png, video: https://youtu.be/3QG9YRsBJaw
(Sorry about the crap quality, but one can clearly see encoding lag out of control and the fact that in some of these the Video Engine Load is pegged at 100% and GPU is not even at 50%)

As you can see from the videos, whenever the encoding lag starts to creep up the Task Manager view of the Encode utilization switches from very stable to spotty and ultimately stops being stable 70% but rather jumps between 50 and 70%, which I feel is the only thing other than GPU-Z GPU Load spike that indicates an encoding lag.

Spec:
- AMD 5800X3D
- 64GB of 3600 CL16 DDR4 RAM (G.Skill Trident Z)
- RTX 4090 FE
- Corsair RM750X 2018 (a weak link, yes, but don't dismiss it for it)
- Fractal Meshify 2 with 9 fans (no lack of cooling)
- RAID0 Intel 660p 2TBx2 SSDs (hardware raid, AMD board. Does not change if it's recorded to a 970 Evo 1TB direct access) for the recording destination
- Both mild overclock (98.7% TimeSpy Stress Test) AND Stock. OC for PS5, Stock for Witcher.

More logs:
- Witcher PC run:

- Valhalla PS5 run:

I had to roll back to the last version because 29 has been destroying my mic quality and causing a lot of audio glitches. This was happening with 2 different PCs, mind you. It's a piece of trash. The expander filter "pops" every time it attacks now. It's just awful. And the upward compressor would be nice if it weren't a hot mess right now.