OBS NVENC encoding problem with RTX 5080

[Otaquet_]

System specs:

• GPU: MSI RTX 5080 (brand new replacement)
• CPU: Intel Core i7-14700KF
• RAM: 64 GB DDR5 (Kingston FURY 6000 MHz)
• Motherboard: ASUS TUF Gaming Z790
• PSU: be quiet! Pure Power 12M – 1000W
• OS: Windows 11 Pro (64-bit)
• NVIDIA Driver: 32.0.15.8157 (Oct 9, 2025)
• OBS Studio: latest stable version
  

Problem description​

Even after upgrading from a 5070 ti to the RTX 5080 , OBS keeps showing “Encoder overloaded” warnings and the GPU usage spikes to 100% while streaming, even with light games or low settings. (like Dune awakening, Battlefront II, Skate)
CPU usage stays around 40%, so it’s not a CPU bottleneck.

OBS stats:

• Average render time: ~20ms
• Missed frames (render): ~4%
• Skipped frames (encoding): 10–15%
• FPS fluctuates between 20–50 instead of stable 60.

Troubleshooting done​

• Clean reinstall of NVIDIA drivers (using DDU).
• Tested different OBS versions (stable + beta).
• Tried NVENC H.264 / HEVC, Quality and Performance presets.
• Different bitrates (6000–10000 kbps) and resolutions (720p–1080p).
• Disabled G-Sync, V-Sync, Ray Tracing, DLSS, and in-game overlays.
• Temps are fine (GPU < 60°C, CPU < 70°C).

Issue persists​

Even with a brand-new RTX 5080, OBS + NVENC cause stutters, frame drops, and high GPU usage on any game.
Other streamers with RTX 50-series cards report similar issues, so it might be a driver or NVENC compatibility problem.

Request​

Could you please confirm:

1. If this is a known issue with RTX 50-series and OBS/NVENC?
2. Any recommended driver version or settings to fix it?
3. If needed, I can provide OBS logs, DxDiag, and GPU-Z info.

Thank you for your help!

---

 

[qhobbes]

Provide a log.

 

[Otaquet_]

Hi

https://obsproject.com/logs/FfgsKHB1O6hDG48v

Here is 3 printscreen from 3 different game I played during the last test.

 

[rockbottom]

Task Manager clearly shows that your GPU is overloaded & the log confirms it as Rendering Lag is GPU overload. Simple stuff. Eliminate the rendering lag & that encoding lag will cease as well. See the link below.

18:07:03.344: Output 'rtmp multitrack video': Total frames output: 660000
18:07:03.344: Output 'rtmp multitrack video': Total drawn frames: 216007 (220135 attempted)
18:07:03.344: Output 'rtmp multitrack video': Number of lagged frames due to rendering lag/stalls: 4128 (1.9%)
18:07:03.344: [rtmp stream: 'rtmp multitrack video'] Freeing 3 remaining packets
18:07:03.344: Video stopped, number of skipped frames due to encoding lag: 4207/220059 (1.9%)
18:07:03.347: ==== Streaming Stop ================================================
18:07:03.392: Video stopped, number of skipped frames due to encoding lag: 4178/220061 (1.9%)
18:07:03.429: Video stopped, number of skipped frames due to encoding lag: 4128/220064 (1.9%)
18:07:03.463: Video stopped, number of skipped frames due to encoding lag: 4128/220066 (1.9%)

I also noticed that your NIC either needs a driver update or you're using Energy Efficient Ethernet settings.
17:05:54.467: [rtmp stream: 'rtmp multitrack video'] Interface: Intel(R) Ethernet Controller I226-V (ethernet, 2500↓/2500↑ mbps)
17:05:54.467: [rtmp stream: 'rtmp multitrack video'] Interface has non-zero error counters (38/0 errors, 38/0 discards)

Encoding Performance Troubleshooting | OBS

OBS Knowledge Base. Learn best practices to solve encoding performance issues

obsproject.com

 

[rockbottom]

You'll free up a significant amount of VRAM by disabling acceleration on your Media Sources. Too many are using it, I stopped counting @ 12. There should only be 2-3 of them using it & only if they're in your main Scene.

17:00:08.531: [Media Source 'Guignols t es un vrai con']: settings:
17:00:08.531: input: C:/Users/User/Desktop/Stream/video son montage/video/DIVERS FR/t es un vrai con euh consommateur j veux dire.mp4
17:00:08.531: input_format: (null)
17:00:08.531: speed: 100
17:00:08.531: is_looping: no
17:00:08.531: is_linear_alpha: no
17:00:08.531: is_hw_decoding: yes
17:00:08.531: is_clear_on_media_end: yes
17:00:08.531: restart_on_activate: yes
17:00:08.531: close_when_inactive: yes
17:00:08.531: full_decode: no

 

[Otaquet_]

hi thanks for you answer
where do I found the "disabling acceleration on your Media Sources." I dont see it on the box to uncheck (it s probably different in french). Which "line" is it under the "local file" section?

 

[rockbottom]

Use Hardware Decoding when ..., 3rd box under the local file.