Question / Help NVENC Intermittently Not Working [RESOLVED]

EposVox

Member
I've been using OBS/OBS Studio w/ Nvenc for YEARS with literally no problems until about a week ago. In the middle of a normal work day, no updates to my PC, no driver updates, no OBS updates - just re-opened it (probably third time in the day I had used it) and tried recording my usual Nvenc profile and it says:
"Starting the output failed. Please check the log for details. Note: If you are using the NVENC or AMD encoders, make sure your drivers are up to date."

Here's the log after it first happened.
Here's my most recent log.
Here's a log of me trying 32-bit, too.

I guess the big thing to note at the bottom:
"
  1. 16:30:02.763: error: OpenEncodeSessionEx failed: out of memory (10)
  2. 16:30:02.790: fatal: No NVENC capable devices found
  3. 16:30:02.790: [NVENC encoder: 'recording_h264'] Failed to open NVENC codec: Generic error in an external library
"

In trying to fix this, I updated Nvidia drivers. Didn't work. Re-installed OBS and it worked for the first day, but the problem keeps happening intermittently. I can record Nvenc streams for hours on end one day, then the next it won't let me record at all. X264 works, but I don't want to use that, especially for a lot of how I use it.

Not sure why it's "out of memory."
Nvence Capable Device is obviously present. Shadowplay/Share itself works (and yes I have it disabled while trying to use OBS) and Adobe CC suite detects the GPU for CUDA.

Any ideas??

Specs:
i7-6900k
32GB RAM
GTX 1080
Windows 10 64-bit (not on creator's update yet)
Oh and this can happen in both 32bit and 64bit OBS, too. Tried that.

2 different Nvenc profiles I usually use:
1080p 60fps 30mbps for stream VODs
&
4k 60fps Lossless for desktop recording
 
Last edited:

Fenrir

Forum Admin
Saw this earlier, and I've been looking in to it, but not able to find much information on that error, outside it's coming from NVENC itself. Just want to let you know I'm still checking in to it for you. Thanks for the detailed report.
 

EposVox

Member
Thanks for letting me know. I figured it might be a Nvidia issue, but since it happened mid-use (not following a driver update or anything) and I haven't had any other GPU symptoms, wanted to see if an OBS solution was possible. Nvenc recording is a key part of my daily operation and ShadowPlay sucks haha.
I guess I could try running Prores via OBS again.
 
Top