OBS NVENC encoding problem with RTX 5080

Otaquet_

New Member
System specs:
  • GPU: MSI RTX 5080 (brand new replacement)
  • CPU: Intel Core i7-14700KF
  • RAM: 64 GB DDR5 (Kingston FURY 6000 MHz)
  • Motherboard: ASUS TUF Gaming Z790
  • PSU: be quiet! Pure Power 12M – 1000W
  • OS: Windows 11 Pro (64-bit)
  • NVIDIA Driver: 32.0.15.8157 (Oct 9, 2025)
  • OBS Studio: latest stable version

Problem description​

Even after upgrading from a 5070 ti to the RTX 5080 , OBS keeps showing “Encoder overloaded” warnings and the GPU usage spikes to 100% while streaming, even with light games or low settings. (like Dune awakening, Battlefront II, Skate)
CPU usage stays around 40%, so it’s not a CPU bottleneck.

OBS stats:
  • Average render time: ~20ms
  • Missed frames (render): ~4%
  • Skipped frames (encoding): 10–15%
  • FPS fluctuates between 20–50 instead of stable 60.

Troubleshooting done​

  • Clean reinstall of NVIDIA drivers (using DDU).
  • Tested different OBS versions (stable + beta).
  • Tried NVENC H.264 / HEVC, Quality and Performance presets.
  • Different bitrates (6000–10000 kbps) and resolutions (720p–1080p).
  • Disabled G-Sync, V-Sync, Ray Tracing, DLSS, and in-game overlays.
  • Temps are fine (GPU < 60°C, CPU < 70°C).

Issue persists​

Even with a brand-new RTX 5080, OBS + NVENC cause stutters, frame drops, and high GPU usage on any game.
Other streamers with RTX 50-series cards report similar issues, so it might be a driver or NVENC compatibility problem.


Request​

Could you please confirm:
  1. If this is a known issue with RTX 50-series and OBS/NVENC?
  2. Any recommended driver version or settings to fix it?
  3. If needed, I can provide OBS logs, DxDiag, and GPU-Z info.
Thank you for your help!



 
Top