Question / Help NVENC can't even do 720p30

[Luk]

Hello everyone!

It's been quite a while since I've last posted here, but I've been having problems with OBS' NVENC feature for a while now.

Previously it was very well possible to stream/encode 1080p with 60FPS, only the bitrate was a limitation. Shadowplay, based on NVENC, does it very well.

There was a time of about half a year where I didn't touch OBS, let alone stream with it. The problems listed below haven't occured before that break and only started after picking up OBS again (and ofc updating it). I have installed OBS from scratch in an attempt to solve the issue (deleted configs, too). Nothing changed.

Recently I've been streaming Fallout 4 for some friends and I'm using NVENC at 3000-4000 kbit/s (problem occurs no matter what the bitrate is set to) with 720p30 to get a decent image quality using NVENC.

Canvas is at 1080p, downscale to 720p using bicubic scaling.

However, while the game is running at 40-90FPS (no VSYNC), OBS' FPS display at the bottom edge shows "30.00 fps" at some time, then it's "21.xx fps", then it's "24.xx fps" etc.

The preview in OBS notably stutters when the fps-display drops below 30.

Those drops mostly occur indoor when I usually have more than 50 FPS in-game.
When I was trying to stream with 60 fps, OBS barely showed >"48.xx fps". This all happens while the CPU is at ~75% max total, GPU at 99% and GPU VID at about 6%. OBS itself is around 5%.

Shadowplay only displays the FPS in-game, no (buffered) background recording is happening.

Here is the logfile of an NVENC and a X264 stream: https://gist.github.com/18eca39cf094fea2fd24dfc386b138fd

Hardware:
Windows 10 Pro x64 w/ Fall Creators Update
Core i5 3570K @ 4.2GHz
16GB DDR3-1600
MSI GTX 960
Internet: 50Mbit/s Download, 10Mbit/s Upload, connected via LAN

 

[Osiris]

Try enabling vsync to prevent the game from using all of your GPU power. Also make sure Windows 10's Game Mode feature is disabled.

Also the last attempt in your log is x264 instead of NVENC.

 

[DarkSwordsman]

If this helps you at all, I use NVENC with my GTX 970 and i5-3570k @ 4.6 Ghz setup.

In a game like GTA V at max settings (their "ultra" preset or whatever) I usually can't do more than 720p @ 30 fps at 10,000 Kbps (I set it high for YouTube since their compression is ridiculous).

If I try 720p @ 60fps, it can kinda work at about 4000 Kbps, but it still will drop to about 40-50 fps on OBS and the game will stutter a little. Even in a game such as Absolute Drift at 720 @ 60fps, the game will drop to about 50 fps and OBS will sit at around 45 fps.

Considering you're running a 960 instead of a 970, I wouldn't be surprised if it was a hardware issue. For as great at NVENC is, it is also not that great.

If you upgraded to say a 1060 to run your games off of (I definitely understand if you may not have the money for it) and then used the 960 for encoding (if you can do it like that), you may be able to see incredibly improved performance.

 

[Luk]

Well, it doesn't make any sense whatsoever.

I can run Shadowplay with 1080p, 60FPS and 50Mbit/s and the recording itself is smooth as... afterwards. When streaming with OBS on a much smaller resolution, half the FPS and 10% of the bitrate, it stutters. This does not make any sense to me, as long as I don't miss a huge point here.

Thanks @Osiris, gonna check if that does the trick.

 

[TryHD]

Shadowplay is efficient and OBS not. You can't compare them.

 

[DarkSwordsman]

Since Shadowplay is designed to work with only Nvidia GPUs by Nvidia themselves and it's extremely limited in terms of features and flexibility compared to OBS, it makes sense why it runs better.

I will ask, do you use GeForce experience or at least update your driver's kinda-frequently? I suggest ditching GeForce Experience and update to the latest drivers constantly. Try that first and get back to us. It will probably not do anything, but multiple processes accessing the same resource certainly slows down those processes.

Oh, and also disable Windows Game Mode. That does nothing but bad.

 

[TryHD]

the scenes he has in his log speak more like dump OBS and go for Geforce Experience. There you can do very basic overlays too.

 

[battler624]

Disable game mode Luk, you have it enabled.

It needs to be disabled on a per-game basis now.

 

[Luk]

Disable game mode Luk, you have it enabled.

I looked up how to disable it via Google, but all I could find was to disable it in the system settings. The problem with that is, after the Fall Creators Update, it cannot be disabled. It only says it's "available". I tried a registry fix, but there is no way I can check whether it is disabled or not. Can you tell me how to force-disable it?

@DarkSwordsman I always run the latest driver and update it using Geforce Experience. I need it because I regularly use the replay-buffer-recording feature of it (but as I said earlier, I disable any kind of recording when streaming).

After trying @Osiris 's suggestion to limit FPS/enable VSync it appears that stuttering only occurs when the GPU is at ~99% load. After forcing the game to run at 30FPS via the RivaTunerStatisticsServer the stream output was smooth as well. An annoying, but functional workaround.