NVENC Video Encoder has disappeared.

[Tizye]

Hey all.

Noticed a couple weeks ago that in my Output settings, my Video Encoder (GPU: NVIDIA GeForce RTX 5070 Ti) is only showing x264 Software encoding through my CPU (AMD Ryzen 9 5900X 12-Core.) instead of showing through my GPU (GPU: NVIDIA GeForce RTX 5070 Ti.)

I was still on Windows 10, so figured that was my cue to finally upgrade. On Windows 11 now, still not showing up. My OBS build was outdated, so figured maybe I should update that as well. Now I'm on OBS 32.1.1, and it's still not showing up. I figured, hey, I've updated my GPU drivers recently, but let's give em another update just in case. Now my GeForce Game Ready Driver is up-to-date with Version 595.97, and it's still not showing up.

I checked the logs, and starting at Line 81 it seems to call the GPU, and then says
21:59:58.742: D3D11 GPU priority setup failed (not admin?)
That's weird, so I restart it, this time Run as Administrator. Now it says
22:10:14.530: D3D11 GPU priority setup success
which sounds great, except it's still not showing up. I also find on Line 118 - 120 saying
22:10:14.877: [NVENC] Test process failed: session_limit
22:10:14.877: NVENC not supported
22:10:14.877: Failed to initialize module 'obs-nvenc.dll'
Which sounds like it's just a missing file, that maybe didn't install correctly with OBS or my NVIDIA GeForce drivers?

I was getting this same issue on my main OBS build, my separate Portable builds, and now on a clean install completely. I know it's not an incompatibility with my GPU, I've used this GPU for streaming dozens of times with no issue, and this only appeared in the past month or so. I'm at a loss.

 

[Warchamp7]

I've had a couple other friends encounter this recently too, and rolling back their driver is what fixed it.

@Tizye did you ever figure this out?

 

[Tizye]

I've had a couple other friends encounter this recently too, and rolling back their driver is what fixed it.

@Tizye did you ever figure this out?

Nope, never figured it out. I've just been encoding through my CPU since then. Luckily this specific rig hasn't been used for any high intensity shows so I've been able to get away with it, but it is definitely limiting.

Rolling back their video drivers? I'd have to roll them back pretty far at this point, but might be worth a try.

 

[rockbottom]

It appears you have too many concurrent NVENC sessions running on your system.

21:29:19.637: [NVENC] Test process failed: session_limit
21:29:19.638: NVENC not supported

22:10:14.877: [NVENC] Test process failed: session_limit
22:10:14.877: NVENC not supported

 

[Tizye]

It appears you have too many concurrent NVENC sessions running on your system.

21:29:19.637: [NVENC] Test process failed: session_limit
21:29:19.638: NVENC not supported

22:10:14.877: [NVENC] Test process failed: session_limit
22:10:14.877: NVENC not supported

I struggle to think what could be causing session limit. It does this immediately upon restart of my computer.

Even in the past when I would run multiple OBS instances on my PC, it would let me set NVENC on multiple of them, but would only let me begin encoding proper on two of them at max. Now it's not even letting me select, the option simply doesn't exist.

 

[rockbottom]

If there's no other apps using NVENC, my guesses in order would be: driver, driver conflict or hardware failure are on the table. Fire up DDU & see if there's a conflict/rollback to the last driver that worked.

 

[joxrgus_tgd(youtube)]

Hey all.

Noticed a couple weeks ago that in my Output settings, my Video Encoder (GPU: NVIDIA GeForce RTX 5070 Ti) is only showing x264 Software encoding through my CPU (AMD Ryzen 9 5900X 12-Core.) instead of showing through my GPU (GPU: NVIDIA GeForce RTX 5070 Ti.)

I was still on Windows 10, so figured that was my cue to finally upgrade. On Windows 11 now, still not showing up. My OBS build was outdated, so figured maybe I should update that as well. Now I'm on OBS 32.1.1, and it's still not showing up. I figured, hey, I've updated my GPU drivers recently, but let's give em another update just in case. Now my GeForce Game Ready Driver is up-to-date with Version 595.97, and it's still not showing up.

I checked the logs, and starting at Line 81 it seems to call the GPU, and then says
21:59:58.742: D3D11 GPU priority setup failed (not admin?)
That's weird, so I restart it, this time Run as Administrator. Now it says
22:10:14.530: D3D11 GPU priority setup success
which sounds great, except it's still not showing up. I also find on Line 118 - 120 saying
22:10:14.877: [NVENC] Test process failed: session_limit
22:10:14.877: NVENC not supported
22:10:14.877: Failed to initialize module 'obs-nvenc.dll'
Which sounds like it's just a missing file, that maybe didn't install correctly with OBS or my NVIDIA GeForce drivers?

I was getting this same issue on my main OBS build, my separate Portable builds, and now on a clean install completely. I know it's not an incompatibility with my GPU, I've used this GPU for streaming dozens of times with no issue, and this only appeared in the past month or so. I'm at a loss.

hey so i faced this issue twice now, ima just tell all the solutions i know and hope it works, and if u already solved the problem pls share it in here(secretly congrats to u). Ill go with baby steps first and if u have already tried them then skip to the next steps.
1. (very basic i know) update your windows, plus your nvidia drivers and then open obs in admin mode for it to work after restarting the pc/laptop.
2. if the step1 doesnt work, something i tried was using ddu(display driver uninstaller) from the official website (wagnardsoft.com), but very important thing to consider is boot the pc in safe mode(watch a yt video or ask chatgpt how it must be done for ur pc/laptop), which will uninstall ur nvidia drivers an then u can reinstall the latest nvidia drivers from the official website.
3. based upon ur logs in obs.. check if there exists a nvEncodeAPI64.dll in the obs-studio/obs-plugins/64bit and if it doesnt exist then there must have been sthg wrong with the installation. in my case it didnt exist so i copied the dll file from C:Windows\System32\DriverStore\FileRepository(one more subfolder after this, try searching with the search option) and paste it in C:/Program Files\obs-studio\obs-plugins\64bit. Then try to restart pc and open obs in admin mode.
4. after step3 didnt work (lemme repeat that... ) AFTER STEP 3 didnt work and u actually pasted the dll file in the obsplugins/64bit folder, try copying the nvEncodeAPI64.dll and paste it in C:/Windows/System32 folder and restart obs.
for me step3 worked a year ago and step 4 worked today so hope it helps ya.

 

[JethroTV]

Hello,

I had a similar issue of NVENC encoder not being a visible option and was able to resolve it, so thought I would post for anyone finding this in the future --

I had the same log lines as the original post:

[NVENC] Test process failed: session_limit
NVENC not supported

For context, I am on an RTX 4070 Ti in Windows 11.

The root issue turned out to be a background service holding up NVENC, not the driver, Windows, or OBS settings. Driver reinstalls, and a clean OBS install didn't help.

To discover this, I tested outside OBS in cmd prompt. Instal ffmpeg winget install ffmpeg, then run the command:

ffmpeg -f lavfi -i testsrc=duration=2:size=1280x720:rate=30 -c:v h264_nvenc -f null -

Running that show errors: OpenEncodeSessionEx failed: incompatible client key (21). If you have no errors, your issue may be different. I also ran the command: nvidia-smi encodersessions. This showed no encoder activity.

The combination of no activity but NVENC being held was odd.

To fix it, I disabled any non-Microsoft services and startup apps that I thought might hit my GPU on startup via MSConfig. I then restarted and the ffmpeg test passed. Then launching OBS, my NVENC option was back as normal.

I then re-enabled Services one at a time, restarting and re-running the test after each. In my case the culprit was AVerMedia's RECentralService. I dot use Avermedia's software anymore, so I removed it and the issue seems resolved. If you have other capture, streaming, remote-desktop, or VR software installed that is hitting your GPU on startup, disable their services and test for the root cause.

Hope that helps someone encountering this. All the searches I started with blamed drivers -- but that wasn't it.

 

[stephenjoys]

Thank you so much for this!

It happened to me just after the Sept Windows 10 update (which must have changed something the avermedia service needed).

I tried driver rollbacks, ddu, different obs versions, etc. with no luck. My only fix up until now was to do a quick reinstall of my gfx drivers before every stream to get nvenc back.

I use an internal avermedia capture card for my camera but it still works fine in OBS without the RECentralService.

Thanks again JethroTV. Not all heroes wear capes! :)