Bug Report OBS Studio 17 cannot use nvenc encoder, even when ffmpeg is compiled with it

[Ribs]

I cannot get OBS to record or stream using nvenc.

I get the following error:

info: [NVENC encoder: 'simple_h264_stream'] settings:
rate_control: CBR
bitrate: 2500
cqp: 0
keyint: 60
preset: default
profile: main
level: auto
width: 1920
height: 1080
2-pass: true
GPU: 0

[nvenc_h264 @ 0x13c7840] Failed to create nvenc instance
warning: [NVENC encoder: 'simple_h264_stream'] Failed to open NVENC codec: Generic error in an external library

However, when using ffmpeg directly, I am able to encode without an issues:

$ ffmpeg -f x11grab -r 30 -s 1920x1080 -i :0.0 -vcodec h264_nvenc -b:v 4500k ~/test.mkv
ffmpeg version N-82927-gc6c888e Copyright (c) 2000-2016 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.4) 20160609
configuration: --prefix=/home/ribs/ffmpeg_build --pkg-config-flags=--static --extra-cflags=-I/home/ribs/ffmpeg_build/include --extra-ldflags=-L/home/ribs/ffmpeg_build/lib --bindir=/home/ribs/bin --enable-cuda --enable-cuvid --enable-libnpp --extra-cflags=-I../nv_sdk --extra-ldflags=-L../nv_sdk --enable-gpl --enable-libass --enable-libfdk-aac --enable-libx264 --enable-libx265 --enable-nvenc --enable-nonfree
libavutil 55. 43.100 / 55. 43.100
libavcodec 57. 70.100 / 57. 70.100
libavformat 57. 61.100 / 57. 61.100
libavdevice 57. 2.100 / 57. 2.100
libavfilter 6. 68.100 / 6. 68.100
libswscale 4. 3.101 / 4. 3.101
libswresample 2. 4.100 / 2. 4.100
libpostproc 54. 2.100 / 54. 2.100
[x11grab @ 0x2bc7320] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, x11grab, from ':0.0':
Duration: N/A, bitrate: N/A
Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1920x1080, 30 fps, 1000k tbr, 1000k tbn, 1000k tbc
File '/home/ribs/test.mkv' already exists. Overwrite ? [y/N] y
Output #0, matroska, to '/home/ribs/test.mkv':
Metadata:
encoder : Lavf57.61.100
Stream #0:0: Video: h264 (h264_nvenc) (Main) (H264 / 0x34363248), bgr0, 1920x1080, q=-1--1, 4500 kb/s, 30 fps, 1k tbn, 30 tbc
Metadata:
encoder : Lavc57.70.100 h264_nvenc
Side data:
cpb: bitrate max/min/avg: 0/0/4500000 buffer size: 9000000 vbv_delay: -1
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_nvenc))
Press [q] to stop, [?] for help
frame= 141 fps= 88 q=14.0 Lsize= 619kB time=00:00:04.66 bitrate=1086.4kbits/s speed=2.92x
video:617kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.281740%

The obs output isn't super helpful, I don't know where to start looking, and as ffmpeg is able to work, it would suggest the issue is with OBS and not ffmpeg.

Can someone point me in the right direction? Have I overlooked something?

 

[Felipe Vieira]

I have the very same issue.

 

[Ribs]

My issue went away when I switched to Arch - worked first time.

I think it might be a mismatch between the driver version installed and the version of the SDK which is downloaded.

You'd either have to upgrade the driver installed, or ensure you download a old enough SDK to match them match.

But I could be wrong.